2024-03-25T17:35:47.043 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T17:35:47.043 DEBUG:teuthology.run:Teuthology command: teuthology --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621303 --verbose --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi --owner scheduled_amathuri@teuthology -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621303/orig.config.yaml 2024-03-25T17:35:47.084 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T17:35:47.159 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621303 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621303' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi006.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJzS5jxOxGYBkmXRwUv79FYTq0F6mAdD0VcTyIyO6APFUyBkiyxdulhH119XzUV6ATQR9i1XYNrHv24NcvH/MVc= smithi192.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNSz2WXJW1fvNy/bJE8JQNd9/IgUiV02UxqYAAg6OOkO/eljBodBu+CP26wgr8RmCgIKY3bFpC1hVE40eVVCvdc= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T17:35:47.159 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T17:35:47.160 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T17:35:47.160 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T17:35:47.161 INFO:teuthology.task.internal:Checking packages... 2024-03-25T17:35:47.183 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T17:35:47.183 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T17:35:47.183 INFO:teuthology.packaging:ref: None 2024-03-25T17:35:47.183 INFO:teuthology.packaging:tag: None 2024-03-25T17:35:47.183 INFO:teuthology.packaging:branch: main 2024-03-25T17:35:47.183 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T17:35:47.184 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T17:35:47.451 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T17:35:47.452 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T17:35:47.461 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T17:35:47.461 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T17:35:47.476 INFO:teuthology.task.internal:Saving configuration 2024-03-25T17:35:47.491 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T17:35:47.502 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T17:35:47.527 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi006.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621303', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 17:30:57.355892', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJzS5jxOxGYBkmXRwUv79FYTq0F6mAdD0VcTyIyO6APFUyBkiyxdulhH119XzUV6ATQR9i1XYNrHv24NcvH/MVc='} 2024-03-25T17:35:47.552 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621303', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 17:30:57.354215', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNSz2WXJW1fvNy/bJE8JQNd9/IgUiV02UxqYAAg6OOkO/eljBodBu+CP26wgr8RmCgIKY3bFpC1hVE40eVVCvdc='} 2024-03-25T17:35:47.552 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T17:35:47.566 INFO:teuthology.task.internal:roles: ubuntu@smithi006.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T17:35:47.566 INFO:teuthology.task.internal:roles: ubuntu@smithi192.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T17:35:47.566 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T17:35:47.647 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc9539260d0>, signals=[15]) 2024-03-25T17:35:47.649 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T17:35:47.660 INFO:teuthology.task.internal:Opening connections... 2024-03-25T17:35:47.660 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi006.front.sepia.ceph.com 2024-03-25T17:35:47.662 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:35:47.766 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi192.front.sepia.ceph.com 2024-03-25T17:35:47.767 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:35:47.853 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T17:35:47.862 DEBUG:teuthology.orchestra.run.smithi006:> uname -m 2024-03-25T17:35:47.881 INFO:teuthology.orchestra.run.smithi006.stdout:x86_64 2024-03-25T17:35:47.881 DEBUG:teuthology.orchestra.run.smithi006:> cat /etc/os-release 2024-03-25T17:35:47.938 INFO:teuthology.orchestra.run.smithi006.stdout:NAME="CentOS Stream" 2024-03-25T17:35:47.939 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION="9" 2024-03-25T17:35:47.939 INFO:teuthology.orchestra.run.smithi006.stdout:ID="centos" 2024-03-25T17:35:47.939 INFO:teuthology.orchestra.run.smithi006.stdout:ID_LIKE="rhel fedora" 2024-03-25T17:35:47.939 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION_ID="9" 2024-03-25T17:35:47.939 INFO:teuthology.orchestra.run.smithi006.stdout:PLATFORM_ID="platform:el9" 2024-03-25T17:35:47.939 INFO:teuthology.orchestra.run.smithi006.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T17:35:47.939 INFO:teuthology.orchestra.run.smithi006.stdout:ANSI_COLOR="0;31" 2024-03-25T17:35:47.939 INFO:teuthology.orchestra.run.smithi006.stdout:LOGO="fedora-logo-icon" 2024-03-25T17:35:47.939 INFO:teuthology.orchestra.run.smithi006.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T17:35:47.939 INFO:teuthology.orchestra.run.smithi006.stdout:HOME_URL="https://centos.org/" 2024-03-25T17:35:47.939 INFO:teuthology.orchestra.run.smithi006.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T17:35:47.939 INFO:teuthology.orchestra.run.smithi006.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T17:35:47.939 INFO:teuthology.orchestra.run.smithi006.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T17:35:47.940 INFO:teuthology.lock.ops:Updating smithi006.front.sepia.ceph.com on lock server 2024-03-25T17:35:47.959 DEBUG:teuthology.orchestra.run.smithi192:> uname -m 2024-03-25T17:35:47.976 INFO:teuthology.orchestra.run.smithi192.stdout:x86_64 2024-03-25T17:35:47.977 DEBUG:teuthology.orchestra.run.smithi192:> cat /etc/os-release 2024-03-25T17:35:48.033 INFO:teuthology.orchestra.run.smithi192.stdout:NAME="CentOS Stream" 2024-03-25T17:35:48.033 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION="9" 2024-03-25T17:35:48.033 INFO:teuthology.orchestra.run.smithi192.stdout:ID="centos" 2024-03-25T17:35:48.033 INFO:teuthology.orchestra.run.smithi192.stdout:ID_LIKE="rhel fedora" 2024-03-25T17:35:48.033 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION_ID="9" 2024-03-25T17:35:48.033 INFO:teuthology.orchestra.run.smithi192.stdout:PLATFORM_ID="platform:el9" 2024-03-25T17:35:48.033 INFO:teuthology.orchestra.run.smithi192.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T17:35:48.034 INFO:teuthology.orchestra.run.smithi192.stdout:ANSI_COLOR="0;31" 2024-03-25T17:35:48.034 INFO:teuthology.orchestra.run.smithi192.stdout:LOGO="fedora-logo-icon" 2024-03-25T17:35:48.034 INFO:teuthology.orchestra.run.smithi192.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T17:35:48.034 INFO:teuthology.orchestra.run.smithi192.stdout:HOME_URL="https://centos.org/" 2024-03-25T17:35:48.034 INFO:teuthology.orchestra.run.smithi192.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T17:35:48.034 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T17:35:48.034 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T17:35:48.034 INFO:teuthology.lock.ops:Updating smithi192.front.sepia.ceph.com on lock server 2024-03-25T17:35:48.059 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T17:35:48.073 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T17:35:48.085 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T17:35:48.085 DEBUG:teuthology.orchestra.run.smithi006:> test '!' -e /home/ubuntu/cephtest 2024-03-25T17:35:48.088 DEBUG:teuthology.orchestra.run.smithi192:> test '!' -e /home/ubuntu/cephtest 2024-03-25T17:35:48.103 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T17:35:48.114 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T17:35:48.114 DEBUG:teuthology.orchestra.run.smithi006:> test -z $(ls -A /var/lib/ceph) 2024-03-25T17:35:48.145 DEBUG:teuthology.orchestra.run.smithi192:> test -z $(ls -A /var/lib/ceph) 2024-03-25T17:35:48.172 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T17:35:48.230 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T17:35:48.248 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T17:35:48.248 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T17:35:48.248 DEBUG:teuthology.orchestra.run.smithi006:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T17:35:48.249 DEBUG:teuthology.orchestra.run.smithi192:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T17:35:48.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:35:48.264 DEBUG:teuthology.orchestra.run.smithi192:> uname -r 2024-03-25T17:35:48.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:35:48.266 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2024-03-25T17:35:48.319 INFO:teuthology.orchestra.run.smithi192.stdout:5.14.0-361.el9.x86_64 2024-03-25T17:35:48.319 INFO:teuthology.task.kernel:Running kernel on smithi192: 5.14.0-361.el9.x86_64 2024-03-25T17:35:48.319 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum install -y kernel 2024-03-25T17:35:48.323 INFO:teuthology.orchestra.run.smithi006.stdout:5.14.0-361.el9.x86_64 2024-03-25T17:35:48.324 INFO:teuthology.task.kernel:Running kernel on smithi006: 5.14.0-361.el9.x86_64 2024-03-25T17:35:48.324 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum install -y kernel 2024-03-25T17:35:50.677 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS-9 - AppStream 9.3 MB/s | 6.7 MB 00:00 2024-03-25T17:35:50.707 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-25T17:35:52.136 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS-9 - BaseOS 6.7 MB/s | 2.0 MB 00:00 2024-03-25T17:35:52.148 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS-9 - BaseOS 6.7 MB/s | 2.0 MB 00:00 2024-03-25T17:36:45.721 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - BaseOS 155 kB/s | 8.0 MB 00:53 2024-03-25T17:36:46.416 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - BaseOS 153 kB/s | 8.0 MB 00:53 2024-03-25T17:36:54.369 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - AppStream 2.5 MB/s | 19 MB 00:07 2024-03-25T17:36:54.954 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - AppStream 2.5 MB/s | 19 MB 00:07 2024-03-25T17:37:06.338 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - CRB 762 kB/s | 6.1 MB 00:08 2024-03-25T17:37:06.726 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - CRB 759 kB/s | 6.1 MB 00:08 2024-03-25T17:37:07.852 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - Extras packages 48 kB/s | 16 kB 00:00 2024-03-25T17:37:08.447 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 16 kB 00:00 2024-03-25T17:37:09.607 INFO:teuthology.orchestra.run.smithi192.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-03-25T17:37:14.558 INFO:teuthology.orchestra.run.smithi192.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-25T17:37:16.705 INFO:teuthology.orchestra.run.smithi192.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T17:37:16.705 INFO:teuthology.orchestra.run.smithi192.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T17:37:16.793 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T17:37:16.806 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:37:16.806 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2024-03-25T17:37:16.806 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:37:16.807 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-25T17:37:16.807 INFO:teuthology.orchestra.run.smithi192.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-25T17:37:16.807 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-03-25T17:37:16.807 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-25T17:37:16.807 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-25T17:37:16.807 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-25T17:37:16.807 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:37:16.807 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-25T17:37:16.807 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:37:16.807 INFO:teuthology.orchestra.run.smithi192.stdout:Install 4 Packages 2024-03-25T17:37:16.807 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:37:16.807 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 76 M 2024-03-25T17:37:16.807 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 123 M 2024-03-25T17:37:16.807 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-25T17:37:16.996 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:37:16.997 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:37:16.997 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:37:17.143 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:37:17.156 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:37:17.180 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:37:17.212 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:37:17.222 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:37:17.292 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:37:17.349 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:37:17.381 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:37:17.419 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:37:17.816 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:37:17.847 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:37:17.885 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:37:18.135 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:37:18.166 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:37:18.208 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:37:18.296 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:37:18.328 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:37:18.368 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:37:18.650 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:37:18.679 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:37:18.715 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:37:18.716 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:37:18.744 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:37:18.749 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:37:18.857 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:37:18.867 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:37:18.889 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:37:18.893 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:37:18.901 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:37:18.924 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:37:18.934 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:37:18.942 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:37:19.003 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:37:19.023 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:37:19.024 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:37:19.043 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:37:19.070 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:37:19.070 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:37:19.089 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:37:19.187 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:37:19.188 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:37:19.210 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:37:19.319 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:37:19.350 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:37:19.351 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:37:19.374 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:19.377 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:37:19.383 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:37:19.411 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:37:19.420 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:19.443 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:19.467 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:19.510 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:19.516 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:37:19.522 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:19.558 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:37:19.558 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:37:19.647 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:37:19.647 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:37:19.684 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:37:19.890 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:37:19.903 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:37:19.925 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:37:20.012 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:37:20.028 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:37:20.033 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:37:20.039 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:37:20.048 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:37:20.106 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:37:20.131 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:37:20.204 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:20.205 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:37:20.238 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:37:20.272 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:20.272 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:20.309 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:20.309 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:20.349 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:20.433 INFO:teuthology.orchestra.run.smithi192.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 19 kB/s | 66 kB 00:03 2024-03-25T17:37:20.600 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:37:20.700 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:37:20.966 INFO:teuthology.orchestra.run.smithi192.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 3.7 MB/s | 15 MB 00:04 2024-03-25T17:37:21.020 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:37:21.143 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:37:21.467 INFO:teuthology.orchestra.run.smithi192.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 7.4 MB/s | 33 MB 00:04 2024-03-25T17:37:21.714 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:37:22.270 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:37:23.785 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:37:23.955 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:37:24.089 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:37:24.160 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:37:24.201 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:37:24.237 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:37:24.283 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:37:24.377 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:37:24.505 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:37:24.600 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:37:24.623 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:24.644 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:37:24.676 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:37:24.730 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:24.878 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:37:24.955 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:37:24.975 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:37:25.064 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:37:25.140 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:25.791 INFO:teuthology.orchestra.run.smithi192.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 5.1 MB/s | 27 MB 00:05 2024-03-25T17:37:25.798 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:37:25.798 INFO:teuthology.orchestra.run.smithi192.stdout:Total 8.4 MB/s | 76 MB 00:08 2024-03-25T17:37:26.455 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-25T17:37:26.813 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-25T17:37:26.813 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-25T17:37:27.834 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-25T17:37:27.834 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-25T17:37:29.329 INFO:teuthology.orchestra.run.smithi006.stdout:Extra Packages for Enterprise Linux 992 kB/s | 21 MB 00:21 2024-03-25T17:37:30.324 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-25T17:37:30.933 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-25T17:37:31.017 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:37:33.767 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:37:33.894 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:37:34.393 INFO:teuthology.orchestra.run.smithi006.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-03-25T17:37:36.642 INFO:teuthology.orchestra.run.smithi006.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T17:37:36.642 INFO:teuthology.orchestra.run.smithi006.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T17:37:36.736 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T17:37:36.742 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:37:36.742 INFO:teuthology.orchestra.run.smithi006.stdout: Package Architecture Version Repository Size 2024-03-25T17:37:36.743 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:37:36.743 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-25T17:37:36.743 INFO:teuthology.orchestra.run.smithi006.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-25T17:37:36.743 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-03-25T17:37:36.743 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-25T17:37:36.743 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-25T17:37:36.743 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-25T17:37:36.743 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:37:36.743 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-25T17:37:36.743 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:37:36.743 INFO:teuthology.orchestra.run.smithi006.stdout:Install 4 Packages 2024-03-25T17:37:36.743 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:37:36.744 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 76 M 2024-03-25T17:37:36.744 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 123 M 2024-03-25T17:37:36.744 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-25T17:37:37.099 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:37:37.099 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:37:37.117 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:37:37.118 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:37:37.118 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:37:37.136 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:37:37.152 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:37:37.152 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:37:37.170 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:37:37.199 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:37:37.199 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:37:37.217 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:37:37.325 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:37:37.325 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:37:37.348 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:37:37.486 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:37:37.487 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:37:37.509 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:37:37.845 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:37:37.865 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:37:37.891 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:37:37.911 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:37:37.930 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:37:37.995 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:37:38.059 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:37:38.101 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:37:38.101 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:37:38.122 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:37:38.147 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:37:38.191 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:37:38.213 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:37:38.213 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:37:38.231 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:38.231 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:38.233 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:37:38.310 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:38.312 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:38.326 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:38.370 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:38.590 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:37:38.702 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:37:38.781 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:37:38.819 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:37:38.846 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:37:39.116 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:37:39.140 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:37:39.171 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:37:39.181 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:37:39.205 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:37:39.263 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:37:39.322 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:37:39.330 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:37:39.389 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:37:39.398 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:37:39.405 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:37:39.485 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:37:39.549 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:37:39.561 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:37:39.636 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:37:39.701 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:37:39.739 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:37:39.776 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:37:39.789 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:37:39.811 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:37:39.815 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:37:39.857 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:37:39.864 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:37:39.893 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:37:39.903 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:37:39.938 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:37:39.938 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:37:39.954 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:37:40.023 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:37:40.024 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:37:40.060 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:37:40.060 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:37:40.103 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:40.103 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:37:40.139 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:40.139 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:40.178 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:40.179 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:40.222 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:40.286 INFO:teuthology.orchestra.run.smithi006.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 20 kB/s | 66 kB 00:03 2024-03-25T17:37:40.408 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:37:40.539 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:37:40.650 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:37:40.686 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:37:40.887 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:37:41.278 INFO:teuthology.orchestra.run.smithi006.stdout:(2/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 7.7 MB/s | 33 MB 00:04 2024-03-25T17:37:41.306 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:37:41.528 INFO:teuthology.orchestra.run.smithi006.stdout:(3/4): kernel-core-5.14.0-432.el9.x86_64.rpm 3.2 MB/s | 15 MB 00:04 2024-03-25T17:37:41.529 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:37:41.890 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:37:41.931 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:37:42.069 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:37:42.099 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:42.189 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:37:42.683 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:37:42.740 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:42.805 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:37:43.131 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:37:43.206 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:37:43.333 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:37:43.416 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:37:43.475 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:37:43.617 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:37:43.644 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:37:43.732 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:37:43.766 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:37:43.845 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:37:43.918 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:44.920 INFO:teuthology.orchestra.run.smithi006.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 5.9 MB/s | 27 MB 00:04 2024-03-25T17:37:44.926 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:37:44.926 INFO:teuthology.orchestra.run.smithi006.stdout:Total 9.3 MB/s | 76 MB 00:08 2024-03-25T17:37:45.613 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-25T17:37:46.031 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-25T17:37:46.031 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-25T17:37:47.070 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-25T17:37:47.071 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-25T17:37:49.678 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-25T17:37:50.311 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-25T17:37:50.396 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:37:53.290 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:37:53.433 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:37:58.510 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:37:58.550 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:38:03.646 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:38:12.492 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:38:12.492 INFO:teuthology.orchestra.run.smithi192.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T17:38:12.492 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:38:12.537 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:38:12.921 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:38:12.921 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-25T17:38:12.921 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:38:12.921 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:38:13.362 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:38:13.362 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:38:13.362 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-25T17:38:13.362 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-25T17:38:13.362 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-25T17:38:13.362 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-25T17:38:13.363 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-25T17:38:13.363 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:38:13.363 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T17:38:13.733 DEBUG:teuthology.orchestra.run.smithi192:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T17:38:13.832 INFO:teuthology.orchestra.run.smithi192.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T17:38:13.833 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T17:38:13.833 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-25T17:38:13.833 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T17:38:13.833 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T17:38:13.833 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T17:38:13.833 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi192.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T17:38:13.833 DEBUG:teuthology.orchestra.run.smithi192:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T17:38:13.913 INFO:teuthology.orchestra.run.smithi192.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T17:38:13.913 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T17:38:13.913 DEBUG:teuthology.orchestra.run.smithi192:> sudo rpm -qi grub2-tools 2024-03-25T17:38:13.957 INFO:teuthology.orchestra.run.smithi192.stdout:Name : grub2-tools 2024-03-25T17:38:13.957 INFO:teuthology.orchestra.run.smithi192.stdout:Epoch : 1 2024-03-25T17:38:13.957 INFO:teuthology.orchestra.run.smithi192.stdout:Version : 2.06 2024-03-25T17:38:13.957 INFO:teuthology.orchestra.run.smithi192.stdout:Release : 68.el9 2024-03-25T17:38:13.957 INFO:teuthology.orchestra.run.smithi192.stdout:Architecture: x86_64 2024-03-25T17:38:13.957 INFO:teuthology.orchestra.run.smithi192.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T17:38:13.957 INFO:teuthology.orchestra.run.smithi192.stdout:Group : Unspecified 2024-03-25T17:38:13.957 INFO:teuthology.orchestra.run.smithi192.stdout:Size : 8264254 2024-03-25T17:38:13.957 INFO:teuthology.orchestra.run.smithi192.stdout:License : GPLv3+ 2024-03-25T17:38:13.958 INFO:teuthology.orchestra.run.smithi192.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T17:38:13.958 INFO:teuthology.orchestra.run.smithi192.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T17:38:13.958 INFO:teuthology.orchestra.run.smithi192.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T17:38:13.958 INFO:teuthology.orchestra.run.smithi192.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T17:38:13.958 INFO:teuthology.orchestra.run.smithi192.stdout:Packager : builder@centos.org 2024-03-25T17:38:13.958 INFO:teuthology.orchestra.run.smithi192.stdout:Vendor : CentOS 2024-03-25T17:38:13.958 INFO:teuthology.orchestra.run.smithi192.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T17:38:13.958 INFO:teuthology.orchestra.run.smithi192.stdout:Summary : Support tools for GRUB. 2024-03-25T17:38:13.958 INFO:teuthology.orchestra.run.smithi192.stdout:Description : 2024-03-25T17:38:13.958 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:38:13.958 INFO:teuthology.orchestra.run.smithi192.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T17:38:13.958 INFO:teuthology.orchestra.run.smithi192.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T17:38:13.958 INFO:teuthology.orchestra.run.smithi192.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T17:38:13.958 INFO:teuthology.orchestra.run.smithi192.stdout:hardware devices. 2024-03-25T17:38:13.958 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:38:13.959 INFO:teuthology.orchestra.run.smithi192.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T17:38:13.959 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T17:38:13.959 INFO:teuthology.task.kernel:Updating grub on smithi192 to boot 5.14.0-432.el9.x86_64 2024-03-25T17:38:13.959 DEBUG:teuthology.orchestra.run.smithi192:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T17:38:14.579 INFO:teuthology.orchestra.run.smithi192.stderr:Generating grub configuration file ... 2024-03-25T17:38:15.876 INFO:teuthology.orchestra.run.smithi192.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T17:38:15.920 INFO:teuthology.orchestra.run.smithi192.stderr:done 2024-03-25T17:38:15.922 DEBUG:teuthology.orchestra.run.smithi192:> mktemp 2024-03-25T17:38:15.937 INFO:teuthology.orchestra.run.smithi192.stdout:/tmp/tmp.RmwgviEDqw 2024-03-25T17:38:15.937 DEBUG:teuthology.orchestra.run.smithi192:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.RmwgviEDqw 2024-03-25T17:38:16.005 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0666 /tmp/tmp.RmwgviEDqw 2024-03-25T17:38:16.138 DEBUG:teuthology.orchestra.remote:smithi192:/tmp/tmp.RmwgviEDqw is 6KB 2024-03-25T17:38:16.185 DEBUG:teuthology.orchestra.run.smithi192:> rm -fr /tmp/tmp.RmwgviEDqw 2024-03-25T17:38:16.199 DEBUG:teuthology.orchestra.run.smithi192:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T17:38:16.264 INFO:teuthology.orchestra.run.smithi192.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T17:38:16.264 INFO:teuthology.orchestra.run.smithi192.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T17:38:16.264 INFO:teuthology.orchestra.run.smithi192.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T17:38:16.264 INFO:teuthology.orchestra.run.smithi192.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-25T17:38:16.265 DEBUG:teuthology.orchestra.run.smithi192:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-25T17:38:16.409 DEBUG:teuthology.orchestra.run.smithi192:> sudo shutdown -r now 2024-03-25T17:38:34.433 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:38:34.433 INFO:teuthology.orchestra.run.smithi006.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T17:38:34.433 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:38:34.479 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:38:34.856 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:38:34.856 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-25T17:38:34.856 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:38:34.856 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:38:35.296 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:38:35.296 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:38:35.297 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-25T17:38:35.297 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-25T17:38:35.297 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-25T17:38:35.297 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-25T17:38:35.297 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-25T17:38:35.297 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:38:35.297 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T17:38:35.618 DEBUG:teuthology.orchestra.run.smithi006:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T17:38:35.725 INFO:teuthology.orchestra.run.smithi006.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T17:38:35.725 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T17:38:35.725 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-25T17:38:35.725 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T17:38:35.725 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T17:38:35.725 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T17:38:35.725 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi006.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T17:38:35.725 DEBUG:teuthology.orchestra.run.smithi006:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T17:38:35.808 INFO:teuthology.orchestra.run.smithi006.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T17:38:35.808 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T17:38:35.808 DEBUG:teuthology.orchestra.run.smithi006:> sudo rpm -qi grub2-tools 2024-03-25T17:38:35.845 INFO:teuthology.orchestra.run.smithi006.stdout:Name : grub2-tools 2024-03-25T17:38:35.846 INFO:teuthology.orchestra.run.smithi006.stdout:Epoch : 1 2024-03-25T17:38:35.846 INFO:teuthology.orchestra.run.smithi006.stdout:Version : 2.06 2024-03-25T17:38:35.846 INFO:teuthology.orchestra.run.smithi006.stdout:Release : 68.el9 2024-03-25T17:38:35.846 INFO:teuthology.orchestra.run.smithi006.stdout:Architecture: x86_64 2024-03-25T17:38:35.846 INFO:teuthology.orchestra.run.smithi006.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T17:38:35.846 INFO:teuthology.orchestra.run.smithi006.stdout:Group : Unspecified 2024-03-25T17:38:35.846 INFO:teuthology.orchestra.run.smithi006.stdout:Size : 8264254 2024-03-25T17:38:35.846 INFO:teuthology.orchestra.run.smithi006.stdout:License : GPLv3+ 2024-03-25T17:38:35.846 INFO:teuthology.orchestra.run.smithi006.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T17:38:35.846 INFO:teuthology.orchestra.run.smithi006.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T17:38:35.846 INFO:teuthology.orchestra.run.smithi006.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T17:38:35.846 INFO:teuthology.orchestra.run.smithi006.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T17:38:35.846 INFO:teuthology.orchestra.run.smithi006.stdout:Packager : builder@centos.org 2024-03-25T17:38:35.847 INFO:teuthology.orchestra.run.smithi006.stdout:Vendor : CentOS 2024-03-25T17:38:35.847 INFO:teuthology.orchestra.run.smithi006.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T17:38:35.847 INFO:teuthology.orchestra.run.smithi006.stdout:Summary : Support tools for GRUB. 2024-03-25T17:38:35.847 INFO:teuthology.orchestra.run.smithi006.stdout:Description : 2024-03-25T17:38:35.847 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:38:35.847 INFO:teuthology.orchestra.run.smithi006.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T17:38:35.847 INFO:teuthology.orchestra.run.smithi006.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T17:38:35.847 INFO:teuthology.orchestra.run.smithi006.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T17:38:35.847 INFO:teuthology.orchestra.run.smithi006.stdout:hardware devices. 2024-03-25T17:38:35.847 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:38:35.847 INFO:teuthology.orchestra.run.smithi006.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T17:38:35.848 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T17:38:35.848 INFO:teuthology.task.kernel:Updating grub on smithi006 to boot 5.14.0-432.el9.x86_64 2024-03-25T17:38:35.848 DEBUG:teuthology.orchestra.run.smithi006:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T17:38:36.437 INFO:teuthology.orchestra.run.smithi006.stderr:Generating grub configuration file ... 2024-03-25T17:38:37.953 INFO:teuthology.orchestra.run.smithi006.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T17:38:37.996 INFO:teuthology.orchestra.run.smithi006.stderr:done 2024-03-25T17:38:37.999 DEBUG:teuthology.orchestra.run.smithi006:> mktemp 2024-03-25T17:38:38.055 INFO:teuthology.orchestra.run.smithi006.stdout:/tmp/tmp.tEKmJJPJIy 2024-03-25T17:38:38.056 DEBUG:teuthology.orchestra.run.smithi006:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.tEKmJJPJIy 2024-03-25T17:38:38.138 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0666 /tmp/tmp.tEKmJJPJIy 2024-03-25T17:38:38.287 DEBUG:teuthology.orchestra.remote:smithi006:/tmp/tmp.tEKmJJPJIy is 6KB 2024-03-25T17:38:38.335 DEBUG:teuthology.orchestra.run.smithi006:> rm -fr /tmp/tmp.tEKmJJPJIy 2024-03-25T17:38:38.350 DEBUG:teuthology.orchestra.run.smithi006:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T17:38:38.418 INFO:teuthology.orchestra.run.smithi006.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T17:38:38.418 INFO:teuthology.orchestra.run.smithi006.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T17:38:38.418 INFO:teuthology.orchestra.run.smithi006.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T17:38:38.418 INFO:teuthology.orchestra.run.smithi006.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-25T17:38:38.420 DEBUG:teuthology.orchestra.run.smithi006:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-25T17:38:38.583 DEBUG:teuthology.orchestra.run.smithi006:> sudo shutdown -r now 2024-03-25T17:38:46.420 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T17:38:46.420 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-03-25T17:38:46.421 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:39:04.862 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.192 2024-03-25T17:39:08.590 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T17:39:08.590 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-03-25T17:39:08.591 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:39:13.867 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-03-25T17:39:13.868 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:39:16.926 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.192 2024-03-25T17:39:27.134 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.6 2024-03-25T17:39:28.929 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-03-25T17:39:28.929 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:39:36.140 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-03-25T17:39:36.141 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:39:47.358 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.192 2024-03-25T17:40:02.374 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-03-25T17:40:02.375 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:40:02.767 DEBUG:teuthology.orchestra.run.smithi192:> true 2024-03-25T17:40:03.179 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-03-25T17:40:03.179 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-432.el9.x86_64"... 2024-03-25T17:40:03.179 DEBUG:teuthology.orchestra.run.smithi192:> uname -r 2024-03-25T17:40:03.193 INFO:teuthology.orchestra.run.smithi192.stdout:5.14.0-432.el9.x86_64 2024-03-25T17:40:03.194 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-25T17:40:03.194 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T17:40:03.194 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T17:40:04.194 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T17:40:04.195 DEBUG:teuthology.orchestra.run.smithi192:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T17:40:04.259 INFO:teuthology.orchestra.run.smithi192.stdout:ttyS1 2024-03-25T17:40:04.299 DEBUG:teuthology.parallel:result is None 2024-03-25T17:40:36.143 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T17:40:48.144 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-03-25T17:40:48.145 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:40:48.495 DEBUG:teuthology.orchestra.run.smithi006:> true 2024-03-25T17:40:48.904 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-03-25T17:40:48.904 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-25T17:40:48.904 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2024-03-25T17:40:48.920 INFO:teuthology.orchestra.run.smithi006.stdout:5.14.0-432.el9.x86_64 2024-03-25T17:40:48.920 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-25T17:40:48.920 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T17:40:48.920 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T17:40:49.921 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T17:40:49.921 DEBUG:teuthology.orchestra.run.smithi006:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T17:40:50.004 INFO:teuthology.orchestra.run.smithi006.stdout:ttyS1 2024-03-25T17:40:50.046 DEBUG:teuthology.parallel:result is None 2024-03-25T17:40:50.047 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T17:40:50.057 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T17:40:50.057 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T17:40:50.060 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T17:40:50.078 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T17:40:50.184 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T17:40:50.195 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T17:40:50.195 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T17:40:50.197 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T17:40:50.416 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T17:40:50.423 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T17:40:50.423 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T17:40:50.425 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T17:40:50.458 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T17:40:50.460 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T17:40:50.472 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T17:40:50.474 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T17:40:50.476 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T17:40:50.482 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T17:40:50.482 DEBUG:teuthology.orchestra.run.smithi006:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T17:40:50.516 DEBUG:teuthology.orchestra.run.smithi192:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T17:40:50.545 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T17:40:50.554 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T17:40:50.554 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T17:40:50.585 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T17:40:50.602 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T17:40:50.679 DEBUG:teuthology.orchestra.run.smithi006:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T17:40:50.733 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T17:40:50.814 DEBUG:teuthology.orchestra.run.smithi006:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T17:40:50.859 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-25T17:40:50.859 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T17:40:50.927 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T17:40:50.971 DEBUG:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T17:40:51.048 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T17:40:51.089 DEBUG:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T17:40:51.134 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-25T17:40:51.134 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T17:40:51.203 DEBUG:teuthology.orchestra.run.smithi006:> sudo service rsyslog restart 2024-03-25T17:40:51.205 DEBUG:teuthology.orchestra.run.smithi192:> sudo service rsyslog restart 2024-03-25T17:40:51.250 INFO:teuthology.orchestra.run.smithi006.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T17:40:51.293 INFO:teuthology.orchestra.run.smithi192.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T17:40:51.683 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T17:40:51.691 INFO:teuthology.task.internal:Starting timer... 2024-03-25T17:40:51.691 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T17:40:51.713 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T17:40:51.721 DEBUG:teuthology.orchestra.run.smithi006:> sudo service auditd rotate 2024-03-25T17:40:51.794 INFO:teuthology.orchestra.run.smithi006.stdout:Rotating logs: 2024-03-25T17:40:51.796 DEBUG:teuthology.orchestra.run.smithi192:> sudo service auditd rotate 2024-03-25T17:40:51.866 INFO:teuthology.orchestra.run.smithi192.stdout:Rotating logs: 2024-03-25T17:40:51.868 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T17:40:51.868 DEBUG:teuthology.orchestra.run.smithi006:> /usr/sbin/getenforce 2024-03-25T17:40:51.892 INFO:teuthology.orchestra.run.smithi006.stdout:Permissive 2024-03-25T17:40:51.892 DEBUG:teuthology.orchestra.run.smithi192:> /usr/sbin/getenforce 2024-03-25T17:40:51.918 INFO:teuthology.orchestra.run.smithi192.stdout:Permissive 2024-03-25T17:40:51.918 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi006.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi192.front.sepia.ceph.com': 'permissive'} 2024-03-25T17:40:51.918 DEBUG:teuthology.orchestra.run.smithi006:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T17:40:51.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:40:51.961 DEBUG:teuthology.orchestra.run.smithi192:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T17:40:51.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:40:51.992 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T17:40:51.993 DEBUG:teuthology.orchestra.run.smithi006:> sudo /usr/sbin/setenforce permissive 2024-03-25T17:40:52.035 DEBUG:teuthology.orchestra.run.smithi192:> sudo /usr/sbin/setenforce permissive 2024-03-25T17:40:52.088 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T17:40:52.103 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T17:40:52.121 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T17:40:52.166 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi006.front.sepia.ceph.com,smithi192.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T17:45:30.453 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi006.front.sepia.ceph.com'), Remote(name='ubuntu@smithi192.front.sepia.ceph.com')] 2024-03-25T17:45:30.454 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-03-25T17:45:30.455 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:45:30.558 DEBUG:teuthology.orchestra.run.smithi006:> true 2024-03-25T17:45:30.643 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-03-25T17:45:30.644 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-03-25T17:45:30.644 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:45:30.731 DEBUG:teuthology.orchestra.run.smithi192:> true 2024-03-25T17:45:30.810 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-03-25T17:45:30.811 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T17:45:30.824 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T17:45:30.824 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T17:45:30.824 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T17:45:30.827 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T17:45:30.827 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T17:45:30.858 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T17:45:30.876 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T17:45:30.884 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T17:45:30.901 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T17:45:30.906 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: ntpd: command not found 2024-03-25T17:45:30.921 INFO:teuthology.orchestra.run.smithi006.stdout:506 Cannot talk to daemon 2024-03-25T17:45:30.931 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: ntpd: command not found 2024-03-25T17:45:30.938 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T17:45:30.945 INFO:teuthology.orchestra.run.smithi192.stdout:506 Cannot talk to daemon 2024-03-25T17:45:30.954 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T17:45:30.962 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T17:45:30.978 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T17:45:31.008 INFO:teuthology.orchestra.run.smithi006.stderr:bash: line 1: ntpq: command not found 2024-03-25T17:45:31.011 INFO:teuthology.orchestra.run.smithi006.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T17:45:31.011 INFO:teuthology.orchestra.run.smithi006.stdout:=============================================================================== 2024-03-25T17:45:31.011 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:45:31.011 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:45:31.011 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:45:31.011 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:45:31.036 INFO:teuthology.orchestra.run.smithi192.stderr:bash: line 1: ntpq: command not found 2024-03-25T17:45:31.039 INFO:teuthology.orchestra.run.smithi192.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T17:45:31.039 INFO:teuthology.orchestra.run.smithi192.stdout:=============================================================================== 2024-03-25T17:45:31.039 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:45:31.039 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:45:31.039 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:45:31.040 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:45:31.041 INFO:teuthology.run_tasks:Running task install... 2024-03-25T17:45:31.049 DEBUG:teuthology.task.install:project ceph 2024-03-25T17:45:31.049 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T17:45:31.049 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T17:45:31.050 INFO:teuthology.task.install:Using flavor: default 2024-03-25T17:45:31.055 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T17:45:31.055 INFO:teuthology.task.install:extra packages: [] 2024-03-25T17:45:31.055 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T17:45:31.056 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T17:45:31.058 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T17:45:31.058 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T17:45:31.203 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T17:45:31.203 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T17:45:31.273 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T17:45:31.273 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T17:45:31.602 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T17:45:31.603 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-25T17:45:31.603 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T17:45:31.633 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T17:45:31.634 DEBUG:teuthology.orchestra.run.smithi006:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T17:45:31.675 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T17:45:31.675 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-25T17:45:31.676 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T17:45:31.705 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T17:45:31.705 DEBUG:teuthology.orchestra.run.smithi192:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T17:45:31.706 DEBUG:teuthology.orchestra.run.smithi006:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T17:45:31.774 DEBUG:teuthology.orchestra.run.smithi192:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T17:45:31.788 DEBUG:teuthology.orchestra.run.smithi006:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T17:45:31.855 DEBUG:teuthology.orchestra.run.smithi192:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T17:45:31.856 INFO:teuthology.orchestra.run.smithi006.stdout:check_obsoletes = 1 2024-03-25T17:45:31.858 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum clean all 2024-03-25T17:45:31.922 INFO:teuthology.orchestra.run.smithi192.stdout:check_obsoletes = 1 2024-03-25T17:45:31.923 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum clean all 2024-03-25T17:45:32.113 INFO:teuthology.orchestra.run.smithi006.stdout:74 files removed 2024-03-25T17:45:32.141 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-radosgw 2024-03-25T17:45:32.171 INFO:teuthology.orchestra.run.smithi192.stdout:74 files removed 2024-03-25T17:45:32.198 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-radosgw 2024-03-25T17:45:33.523 INFO:teuthology.orchestra.run.smithi006.stdout:ceph packages for x86_64 76 kB/s | 82 kB 00:01 2024-03-25T17:45:33.694 INFO:teuthology.orchestra.run.smithi192.stdout:ceph packages for x86_64 68 kB/s | 82 kB 00:01 2024-03-25T17:45:34.678 INFO:teuthology.orchestra.run.smithi192.stdout:ceph noarch packages 11 kB/s | 10 kB 00:00 2024-03-25T17:45:34.832 INFO:teuthology.orchestra.run.smithi006.stdout:ceph noarch packages 8.0 kB/s | 10 kB 00:01 2024-03-25T17:45:35.528 INFO:teuthology.orchestra.run.smithi192.stdout:ceph source packages 2.3 kB/s | 1.9 kB 00:00 2024-03-25T17:45:35.774 INFO:teuthology.orchestra.run.smithi006.stdout:ceph source packages 2.1 kB/s | 1.9 kB 00:00 2024-03-25T17:45:36.005 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-25T17:45:36.210 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-25T17:45:37.648 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-25T17:45:37.802 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS-9 - BaseOS 4.7 MB/s | 2.0 MB 00:00 2024-03-25T17:45:44.804 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - BaseOS 1.2 MB/s | 8.0 MB 00:06 2024-03-25T17:45:46.851 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - BaseOS 948 kB/s | 8.0 MB 00:08 2024-03-25T17:45:54.475 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - AppStream 2.2 MB/s | 19 MB 00:08 2024-03-25T17:45:56.297 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - AppStream 2.3 MB/s | 19 MB 00:08 2024-03-25T17:46:07.179 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - CRB 688 kB/s | 6.1 MB 00:09 2024-03-25T17:46:08.900 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 16 kB 00:00 2024-03-25T17:46:09.192 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - CRB 683 kB/s | 6.1 MB 00:09 2024-03-25T17:46:10.563 INFO:teuthology.orchestra.run.smithi192.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-25T17:46:10.890 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-03-25T17:46:13.002 INFO:teuthology.orchestra.run.smithi006.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:01 2024-03-25T17:46:15.494 INFO:teuthology.orchestra.run.smithi192.stdout:lab-extras 18 kB/s | 1.7 kB 00:00 2024-03-25T17:46:17.957 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T17:46:17.966 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:46:17.967 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-03-25T17:46:17.967 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:46:17.967 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-25T17:46:17.967 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T17:46:17.967 INFO:teuthology.orchestra.run.smithi192.stdout:Upgrading: 2024-03-25T17:46:17.967 INFO:teuthology.orchestra.run.smithi192.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T17:46:17.967 INFO:teuthology.orchestra.run.smithi192.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T17:46:17.967 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-03-25T17:46:17.967 INFO:teuthology.orchestra.run.smithi192.stdout: boost-program-options 2024-03-25T17:46:17.967 INFO:teuthology.orchestra.run.smithi192.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T17:46:17.967 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T17:46:17.967 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:46:17.967 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T17:46:17.968 INFO:teuthology.orchestra.run.smithi192.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T17:46:17.968 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T17:46:17.968 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T17:46:17.968 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T17:46:17.968 INFO:teuthology.orchestra.run.smithi192.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T17:46:17.968 INFO:teuthology.orchestra.run.smithi192.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T17:46:17.968 INFO:teuthology.orchestra.run.smithi192.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T17:46:17.968 INFO:teuthology.orchestra.run.smithi192.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T17:46:17.968 INFO:teuthology.orchestra.run.smithi192.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T17:46:17.968 INFO:teuthology.orchestra.run.smithi192.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T17:46:17.968 INFO:teuthology.orchestra.run.smithi192.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T17:46:17.968 INFO:teuthology.orchestra.run.smithi192.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T17:46:17.968 INFO:teuthology.orchestra.run.smithi192.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T17:46:17.969 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-argparse 2024-03-25T17:46:17.969 INFO:teuthology.orchestra.run.smithi192.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T17:46:17.969 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T17:46:17.969 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T17:46:17.969 INFO:teuthology.orchestra.run.smithi192.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T17:46:17.969 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T17:46:17.969 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T17:46:17.969 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T17:46:17.969 INFO:teuthology.orchestra.run.smithi192.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-25T17:46:17.969 INFO:teuthology.orchestra.run.smithi192.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-25T17:46:17.969 INFO:teuthology.orchestra.run.smithi192.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T17:46:17.969 INFO:teuthology.orchestra.run.smithi192.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T17:46:17.969 INFO:teuthology.orchestra.run.smithi192.stdout:Installing weak dependencies: 2024-03-25T17:46:17.969 INFO:teuthology.orchestra.run.smithi192.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-25T17:46:17.970 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:46:17.970 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-25T17:46:17.970 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:46:17.970 INFO:teuthology.orchestra.run.smithi192.stdout:Install 30 Packages 2024-03-25T17:46:17.970 INFO:teuthology.orchestra.run.smithi192.stdout:Upgrade 2 Packages 2024-03-25T17:46:17.970 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:46:17.970 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 58 M 2024-03-25T17:46:17.970 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-25T17:46:18.308 INFO:teuthology.orchestra.run.smithi006.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-25T17:46:20.904 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T17:46:20.912 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:46:20.912 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-03-25T17:46:20.912 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:46:20.912 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-25T17:46:20.912 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T17:46:20.913 INFO:teuthology.orchestra.run.smithi006.stdout:Upgrading: 2024-03-25T17:46:20.913 INFO:teuthology.orchestra.run.smithi006.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T17:46:20.913 INFO:teuthology.orchestra.run.smithi006.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T17:46:20.913 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-03-25T17:46:20.913 INFO:teuthology.orchestra.run.smithi006.stdout: boost-program-options 2024-03-25T17:46:20.913 INFO:teuthology.orchestra.run.smithi006.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T17:46:20.913 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T17:46:20.913 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:46:20.913 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T17:46:20.913 INFO:teuthology.orchestra.run.smithi006.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T17:46:20.913 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T17:46:20.913 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T17:46:20.913 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T17:46:20.914 INFO:teuthology.orchestra.run.smithi006.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T17:46:20.914 INFO:teuthology.orchestra.run.smithi006.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T17:46:20.914 INFO:teuthology.orchestra.run.smithi006.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T17:46:20.914 INFO:teuthology.orchestra.run.smithi006.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T17:46:20.914 INFO:teuthology.orchestra.run.smithi006.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T17:46:20.914 INFO:teuthology.orchestra.run.smithi006.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T17:46:20.914 INFO:teuthology.orchestra.run.smithi006.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T17:46:20.914 INFO:teuthology.orchestra.run.smithi006.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T17:46:20.914 INFO:teuthology.orchestra.run.smithi006.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T17:46:20.914 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-argparse 2024-03-25T17:46:20.914 INFO:teuthology.orchestra.run.smithi006.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T17:46:20.914 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T17:46:20.914 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T17:46:20.915 INFO:teuthology.orchestra.run.smithi006.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T17:46:20.915 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T17:46:20.915 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T17:46:20.915 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T17:46:20.915 INFO:teuthology.orchestra.run.smithi006.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-25T17:46:20.915 INFO:teuthology.orchestra.run.smithi006.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-25T17:46:20.915 INFO:teuthology.orchestra.run.smithi006.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T17:46:20.915 INFO:teuthology.orchestra.run.smithi006.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T17:46:20.915 INFO:teuthology.orchestra.run.smithi006.stdout:Installing weak dependencies: 2024-03-25T17:46:20.915 INFO:teuthology.orchestra.run.smithi006.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-25T17:46:20.915 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:46:20.915 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-25T17:46:20.915 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:46:20.915 INFO:teuthology.orchestra.run.smithi006.stdout:Install 30 Packages 2024-03-25T17:46:20.915 INFO:teuthology.orchestra.run.smithi006.stdout:Upgrade 2 Packages 2024-03-25T17:46:20.916 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:46:20.916 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 58 M 2024-03-25T17:46:20.916 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-25T17:46:30.627 INFO:teuthology.orchestra.run.smithi192.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 444 kB/s | 5.3 MB 00:12 2024-03-25T17:46:30.836 INFO:teuthology.orchestra.run.smithi192.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 121 kB/s | 25 kB 00:00 2024-03-25T17:46:34.027 INFO:teuthology.orchestra.run.smithi192.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 232 kB/s | 742 kB 00:03 2024-03-25T17:46:36.011 INFO:teuthology.orchestra.run.smithi192.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 251 kB/s | 498 kB 00:01 2024-03-25T17:46:42.869 INFO:teuthology.orchestra.run.smithi192.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 406 kB/s | 9.7 MB 00:24 2024-03-25T17:46:42.941 INFO:teuthology.orchestra.run.smithi006.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 252 kB/s | 5.3 MB 00:21 2024-03-25T17:46:43.341 INFO:teuthology.orchestra.run.smithi006.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 63 kB/s | 25 kB 00:00 2024-03-25T17:46:43.452 INFO:teuthology.orchestra.run.smithi192.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 77 kB/s | 45 kB 00:00 2024-03-25T17:46:44.369 INFO:teuthology.orchestra.run.smithi192.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 143 kB/s | 131 kB 00:00 2024-03-25T17:46:45.720 INFO:teuthology.orchestra.run.smithi192.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 120 kB/s | 162 kB 00:01 2024-03-25T17:46:47.358 INFO:teuthology.orchestra.run.smithi006.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 185 kB/s | 742 kB 00:04 2024-03-25T17:46:48.078 INFO:teuthology.orchestra.run.smithi192.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 136 kB/s | 320 kB 00:02 2024-03-25T17:46:49.128 INFO:teuthology.orchestra.run.smithi192.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 284 kB/s | 298 kB 00:01 2024-03-25T17:46:49.537 INFO:teuthology.orchestra.run.smithi192.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 243 kB/s | 99 kB 00:00 2024-03-25T17:46:49.654 INFO:teuthology.orchestra.run.smithi192.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 908 kB/s | 106 kB 00:00 2024-03-25T17:46:49.704 INFO:teuthology.orchestra.run.smithi192.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-25T17:46:49.737 INFO:teuthology.orchestra.run.smithi192.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T17:46:49.796 INFO:teuthology.orchestra.run.smithi192.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-25T17:46:49.838 INFO:teuthology.orchestra.run.smithi192.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T17:46:49.872 INFO:teuthology.orchestra.run.smithi192.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-25T17:46:49.895 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:46:49.959 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:46:50.119 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:46:50.424 INFO:teuthology.orchestra.run.smithi006.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 163 kB/s | 498 kB 00:03 2024-03-25T17:46:50.469 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:46:50.504 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:46:50.626 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:46:50.645 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:46:50.710 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:46:50.834 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:46:50.876 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:46:50.965 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:46:51.031 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:46:51.161 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:46:51.207 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:46:51.305 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:46:51.777 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:46:52.111 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:46:52.166 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:46:52.201 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:46:52.277 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:46:52.418 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:46:52.579 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:46:52.936 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:46:52.971 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:46:53.050 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:46:53.163 INFO:teuthology.orchestra.run.smithi192.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 67 kB/s | 221 kB 00:03 2024-03-25T17:46:53.181 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:46:53.234 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:46:53.396 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:46:53.753 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:46:53.789 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:46:53.910 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:46:53.928 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:46:53.994 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:46:54.118 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:46:54.159 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:46:54.248 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:46:54.325 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:46:54.458 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:46:54.500 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:46:54.595 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:46:55.080 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:46:55.425 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:46:55.482 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:46:55.502 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:46:55.578 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:46:55.739 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:46:55.900 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:46:56.254 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:46:56.321 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:46:56.427 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:46:56.538 INFO:teuthology.orchestra.run.smithi192.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 11 kB/s | 36 kB 00:03 2024-03-25T17:46:56.556 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:46:56.615 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:46:56.810 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:46:57.195 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:46:57.231 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:46:57.355 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:46:57.374 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:46:57.438 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:46:57.571 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:46:57.610 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:46:57.702 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:46:57.769 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:46:57.900 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:46:57.941 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:46:58.031 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:46:58.511 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:46:58.713 INFO:teuthology.orchestra.run.smithi192.stdout:(20/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 230 kB/s | 5.1 MB 00:22 2024-03-25T17:46:58.814 INFO:teuthology.orchestra.run.smithi192.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 3.1 MB/s | 309 kB 00:00 2024-03-25T17:46:58.867 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:46:58.972 INFO:teuthology.orchestra.run.smithi192.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 28 MB/s | 4.4 MB 00:00 2024-03-25T17:46:58.986 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:46:59.004 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:46:59.022 INFO:teuthology.orchestra.run.smithi192.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 530 kB/s | 26 kB 00:00 2024-03-25T17:46:59.064 INFO:teuthology.orchestra.run.smithi192.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-03-25T17:46:59.115 INFO:teuthology.orchestra.run.smithi192.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-03-25T17:46:59.115 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:46:59.182 INFO:teuthology.orchestra.run.smithi192.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 12 MB/s | 838 kB 00:00 2024-03-25T17:46:59.257 INFO:teuthology.orchestra.run.smithi192.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 3.8 MB/s | 192 kB 00:00 2024-03-25T17:46:59.332 INFO:teuthology.orchestra.run.smithi192.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 21 MB/s | 1.6 MB 00:00 2024-03-25T17:46:59.333 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:46:59.492 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:46:59.843 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:46:59.879 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:46:59.959 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:47:00.049 INFO:teuthology.orchestra.run.smithi192.stdout:(29/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 13 kB/s | 47 kB 00:03 2024-03-25T17:47:04.998 INFO:teuthology.orchestra.run.smithi006.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 228 kB/s | 9.7 MB 00:43 2024-03-25T17:47:05.432 INFO:teuthology.orchestra.run.smithi006.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 104 kB/s | 45 kB 00:00 2024-03-25T17:47:07.182 INFO:teuthology.orchestra.run.smithi006.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 75 kB/s | 131 kB 00:01 2024-03-25T17:47:08.241 INFO:teuthology.orchestra.run.smithi006.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 153 kB/s | 162 kB 00:01 2024-03-25T17:47:10.849 INFO:teuthology.orchestra.run.smithi006.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 123 kB/s | 320 kB 00:02 2024-03-25T17:47:12.065 INFO:teuthology.orchestra.run.smithi192.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 267 kB/s | 3.1 MB 00:12 2024-03-25T17:47:12.366 INFO:teuthology.orchestra.run.smithi006.stdout:(10/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 398 kB/s | 20 MB 00:51 2024-03-25T17:47:12.449 INFO:teuthology.orchestra.run.smithi006.stdout:(11/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 237 kB/s | 5.1 MB 00:22 2024-03-25T17:47:12.558 INFO:teuthology.orchestra.run.smithi006.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 978 kB/s | 106 kB 00:00 2024-03-25T17:47:12.600 INFO:teuthology.orchestra.run.smithi006.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-25T17:47:12.633 INFO:teuthology.orchestra.run.smithi006.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T17:47:12.717 INFO:teuthology.orchestra.run.smithi006.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 661 kB 00:00 2024-03-25T17:47:12.742 INFO:teuthology.orchestra.run.smithi006.stdout:(16/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 157 kB/s | 298 kB 00:01 2024-03-25T17:47:12.776 INFO:teuthology.orchestra.run.smithi006.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.7 MB/s | 282 kB 00:00 2024-03-25T17:47:12.801 INFO:teuthology.orchestra.run.smithi006.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 712 kB/s | 41 kB 00:00 2024-03-25T17:47:12.847 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:47:12.847 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:47:12.893 INFO:teuthology.orchestra.run.smithi006.stdout:(19/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 188 kB/s | 99 kB 00:00 2024-03-25T17:47:12.972 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:47:12.975 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:47:12.997 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:47:12.997 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:47:13.042 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:47:13.065 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:47:13.069 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:47:13.193 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:47:13.201 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:47:13.227 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:47:13.357 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:47:13.358 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:47:13.379 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:47:13.379 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:47:13.397 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:47:13.414 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:47:13.444 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:47:13.444 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:47:13.526 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:47:13.584 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:47:13.595 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:47:13.649 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:47:13.745 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:47:13.760 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:47:13.810 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:47:14.098 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:47:14.121 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:47:14.132 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:47:14.150 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:47:14.155 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:47:14.185 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:47:14.192 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:47:14.226 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:47:14.227 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:47:14.245 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:47:14.245 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:47:14.280 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:47:14.298 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:47:14.299 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:47:14.351 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:47:14.373 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:47:14.373 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:47:14.392 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:47:14.530 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:47:14.531 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:47:14.551 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:47:14.883 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:47:14.907 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:47:14.932 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:47:15.366 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:47:15.402 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:47:15.417 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:47:15.691 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:47:15.727 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:47:15.762 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:47:15.763 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:47:15.797 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:47:15.805 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:47:15.815 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:47:15.815 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:47:15.833 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:47:15.880 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:47:15.891 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:47:15.910 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:47:16.030 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:47:16.043 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:47:16.066 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:47:16.066 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:47:16.077 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:47:16.132 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:47:16.149 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:47:16.149 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:47:16.169 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:47:16.304 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:47:16.314 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:47:16.352 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:47:16.390 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:47:16.393 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:47:16.406 INFO:teuthology.orchestra.run.smithi192.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 202 kB/s | 3.4 MB 00:17 2024-03-25T17:47:16.451 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:47:16.502 INFO:teuthology.orchestra.run.smithi006.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 13 kB/s | 47 kB 00:03 2024-03-25T17:47:16.527 INFO:teuthology.orchestra.run.smithi006.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 9.6 kB/s | 36 kB 00:03 2024-03-25T17:47:16.577 INFO:teuthology.orchestra.run.smithi006.stdout:(22/32): qatlib-24.02.0-1.el9.x86_64.rpm 58 kB/s | 221 kB 00:03 2024-03-25T17:47:16.744 INFO:teuthology.orchestra.run.smithi006.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 183 kB/s | 26 kB 00:00 2024-03-25T17:47:16.778 INFO:teuthology.orchestra.run.smithi006.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.1 MB/s | 309 kB 00:00 2024-03-25T17:47:16.945 INFO:teuthology.orchestra.run.smithi006.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 243 kB/s | 49 kB 00:00 2024-03-25T17:47:16.970 INFO:teuthology.orchestra.run.smithi006.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 404 kB/s | 67 kB 00:00 2024-03-25T17:47:17.145 INFO:teuthology.orchestra.run.smithi006.stdout:(27/32): libarrow-9.0.0-10.el9.x86_64.rpm 7.4 MB/s | 4.4 MB 00:00 2024-03-25T17:47:17.228 INFO:teuthology.orchestra.run.smithi006.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 820 kB/s | 192 kB 00:00 2024-03-25T17:47:17.296 INFO:teuthology.orchestra.run.smithi006.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.3 MB/s | 838 kB 00:00 2024-03-25T17:47:17.379 INFO:teuthology.orchestra.run.smithi006.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 6.8 MB/s | 1.6 MB 00:00 2024-03-25T17:47:26.654 INFO:teuthology.orchestra.run.smithi006.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 342 kB/s | 3.1 MB 00:09 2024-03-25T17:47:27.254 INFO:teuthology.orchestra.run.smithi006.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 345 kB/s | 3.4 MB 00:10 2024-03-25T17:47:27.265 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:47:27.265 INFO:teuthology.orchestra.run.smithi006.stdout:Total 893 kB/s | 58 MB 01:06 2024-03-25T17:47:27.297 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-25T17:47:27.340 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-25T17:47:27.340 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-25T17:47:27.738 INFO:teuthology.orchestra.run.smithi192.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 293 kB/s | 20 MB 01:09 2024-03-25T17:47:27.746 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:47:27.746 INFO:teuthology.orchestra.run.smithi192.stdout:Total 849 kB/s | 58 MB 01:09 2024-03-25T17:47:27.776 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-25T17:47:27.815 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-25T17:47:27.815 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-25T17:47:27.828 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-25T17:47:27.828 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-25T17:47:28.292 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-25T17:47:28.293 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-25T17:47:29.310 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-25T17:47:29.407 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T17:47:29.442 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T17:47:29.475 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T17:47:29.506 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T17:47:29.523 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T17:47:29.683 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T17:47:29.738 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T17:47:29.767 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-25T17:47:29.854 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T17:47:29.893 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T17:47:29.918 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T17:47:29.950 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T17:47:29.966 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T17:47:30.094 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T17:47:30.152 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T17:47:30.158 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T17:47:30.213 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T17:47:30.236 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T17:47:30.326 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T17:47:30.362 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T17:47:30.410 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T17:47:30.453 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T17:47:30.498 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T17:47:30.534 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T17:47:30.567 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T17:47:30.589 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T17:47:30.612 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T17:47:30.637 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T17:47:30.672 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-25T17:47:30.703 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T17:47:30.783 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T17:47:30.792 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T17:47:30.822 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T17:47:30.823 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T17:47:30.864 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T17:47:30.885 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T17:47:30.896 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T17:47:30.913 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T17:47:30.925 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T17:47:30.957 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T17:47:30.958 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T17:47:31.025 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T17:47:31.058 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T17:47:31.122 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T17:47:31.165 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-25T17:47:31.272 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T17:47:31.307 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T17:47:31.315 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T17:47:31.346 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T17:47:31.377 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T17:47:31.405 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T17:47:31.420 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T17:47:31.435 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T17:47:31.451 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T17:47:31.468 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T17:47:31.491 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T17:47:31.795 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T17:47:31.823 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T17:47:31.926 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T17:47:31.953 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T17:47:31.975 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T17:47:31.991 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T17:47:31.998 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T17:47:32.053 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T17:47:32.124 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T17:47:32.167 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T17:47:32.462 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T17:47:32.492 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T17:47:32.547 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T17:47:32.620 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T17:47:32.653 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T17:47:34.481 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T17:47:34.537 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T17:47:35.043 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T17:47:35.091 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T17:47:35.143 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T17:47:35.183 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T17:47:35.278 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T17:47:35.278 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T17:47:35.278 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T17:47:35.278 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:47:35.317 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T17:47:35.637 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T17:47:35.678 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T17:47:35.761 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T17:47:35.762 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T17:47:35.762 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T17:47:35.762 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:47:35.820 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T17:47:45.074 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T17:47:45.075 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /sys 2024-03-25T17:47:45.075 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /proc 2024-03-25T17:47:45.075 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /mnt 2024-03-25T17:47:45.075 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /var/tmp 2024-03-25T17:47:45.075 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /home 2024-03-25T17:47:45.075 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /root 2024-03-25T17:47:45.075 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /tmp 2024-03-25T17:47:45.075 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:47:45.149 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T17:47:45.149 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /sys 2024-03-25T17:47:45.149 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /proc 2024-03-25T17:47:45.150 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /mnt 2024-03-25T17:47:45.150 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /var/tmp 2024-03-25T17:47:45.150 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /home 2024-03-25T17:47:45.150 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /root 2024-03-25T17:47:45.150 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /tmp 2024-03-25T17:47:45.150 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:47:45.947 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T17:47:45.971 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T17:47:45.971 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:47:45.971 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T17:47:45.971 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T17:47:45.971 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T17:47:45.971 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:47:45.973 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T17:47:46.058 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T17:47:46.058 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T17:47:46.077 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T17:47:46.101 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T17:47:46.101 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:47:46.101 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T17:47:46.102 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T17:47:46.102 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T17:47:46.102 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:47:46.103 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T17:47:46.189 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T17:47:46.189 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T17:47:47.205 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T17:47:47.205 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T17:47:47.205 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T17:47:47.205 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T17:47:47.205 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T17:47:47.205 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T17:47:47.205 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T17:47:47.205 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T17:47:47.205 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T17:47:47.205 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T17:47:47.205 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T17:47:47.206 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T17:47:47.206 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T17:47:47.206 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T17:47:47.206 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T17:47:47.206 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T17:47:47.206 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T17:47:47.206 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T17:47:47.206 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T17:47:47.206 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T17:47:47.206 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-25T17:47:47.206 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-25T17:47:47.206 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-25T17:47:47.206 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T17:47:47.206 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T17:47:47.207 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T17:47:47.207 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T17:47:47.207 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T17:47:47.207 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T17:47:47.207 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T17:47:47.207 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T17:47:47.208 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T17:47:47.208 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T17:47:47.208 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T17:47:47.320 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T17:47:47.320 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T17:47:47.320 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T17:47:47.320 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T17:47:47.320 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T17:47:47.320 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T17:47:47.320 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T17:47:47.320 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T17:47:47.320 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T17:47:47.321 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T17:47:47.321 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T17:47:47.321 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T17:47:47.321 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T17:47:47.321 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T17:47:47.321 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T17:47:47.321 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T17:47:47.321 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T17:47:47.321 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T17:47:47.321 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T17:47:47.321 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T17:47:47.321 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-25T17:47:47.321 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-25T17:47:47.321 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-25T17:47:47.321 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T17:47:47.322 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T17:47:47.322 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T17:47:47.322 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T17:47:47.322 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T17:47:47.322 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T17:47:47.322 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T17:47:47.322 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T17:47:47.322 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T17:47:47.322 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T17:47:47.322 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T17:47:48.271 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T17:47:48.271 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:47:48.271 INFO:teuthology.orchestra.run.smithi192.stdout:Upgraded: 2024-03-25T17:47:48.271 INFO:teuthology.orchestra.run.smithi192.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:48.271 INFO:teuthology.orchestra.run.smithi192.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:48.271 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-25T17:47:48.271 INFO:teuthology.orchestra.run.smithi192.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T17:47:48.271 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:48.271 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:48.271 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:48.271 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:48.271 INFO:teuthology.orchestra.run.smithi192.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T17:47:48.272 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T17:47:48.272 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T17:47:48.272 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:48.272 INFO:teuthology.orchestra.run.smithi192.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T17:47:48.272 INFO:teuthology.orchestra.run.smithi192.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T17:47:48.272 INFO:teuthology.orchestra.run.smithi192.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T17:47:48.272 INFO:teuthology.orchestra.run.smithi192.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:48.272 INFO:teuthology.orchestra.run.smithi192.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T17:47:48.272 INFO:teuthology.orchestra.run.smithi192.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:48.272 INFO:teuthology.orchestra.run.smithi192.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T17:47:48.272 INFO:teuthology.orchestra.run.smithi192.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T17:47:48.272 INFO:teuthology.orchestra.run.smithi192.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T17:47:48.272 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:48.272 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:48.273 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:48.273 INFO:teuthology.orchestra.run.smithi192.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T17:47:48.273 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:48.273 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:48.273 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:48.273 INFO:teuthology.orchestra.run.smithi192.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-25T17:47:48.273 INFO:teuthology.orchestra.run.smithi192.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-25T17:47:48.273 INFO:teuthology.orchestra.run.smithi192.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-25T17:47:48.273 INFO:teuthology.orchestra.run.smithi192.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T17:47:48.273 INFO:teuthology.orchestra.run.smithi192.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T17:47:48.273 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:47:48.273 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T17:47:48.390 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T17:47:48.390 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:47:48.391 INFO:teuthology.orchestra.run.smithi006.stdout:Upgraded: 2024-03-25T17:47:48.391 INFO:teuthology.orchestra.run.smithi006.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:48.391 INFO:teuthology.orchestra.run.smithi006.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:48.391 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-25T17:47:48.391 INFO:teuthology.orchestra.run.smithi006.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T17:47:48.391 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:48.391 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:48.391 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:48.391 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:48.391 INFO:teuthology.orchestra.run.smithi006.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T17:47:48.391 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T17:47:48.391 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T17:47:48.391 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:48.391 INFO:teuthology.orchestra.run.smithi006.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T17:47:48.391 INFO:teuthology.orchestra.run.smithi006.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T17:47:48.392 INFO:teuthology.orchestra.run.smithi006.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T17:47:48.392 INFO:teuthology.orchestra.run.smithi006.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:48.392 INFO:teuthology.orchestra.run.smithi006.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T17:47:48.392 INFO:teuthology.orchestra.run.smithi006.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:48.392 INFO:teuthology.orchestra.run.smithi006.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T17:47:48.392 INFO:teuthology.orchestra.run.smithi006.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T17:47:48.392 INFO:teuthology.orchestra.run.smithi006.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T17:47:48.392 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:48.392 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:48.392 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:48.392 INFO:teuthology.orchestra.run.smithi006.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T17:47:48.392 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:48.392 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:48.392 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:48.392 INFO:teuthology.orchestra.run.smithi006.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-25T17:47:48.392 INFO:teuthology.orchestra.run.smithi006.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-25T17:47:48.393 INFO:teuthology.orchestra.run.smithi006.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-25T17:47:48.393 INFO:teuthology.orchestra.run.smithi006.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T17:47:48.393 INFO:teuthology.orchestra.run.smithi006.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T17:47:48.393 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:47:48.393 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T17:47:48.657 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-test 2024-03-25T17:47:48.667 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-test 2024-03-25T17:47:49.221 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:31 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T17:47:49.225 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:34 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T17:47:49.378 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T17:47:49.378 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:47:49.378 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-03-25T17:47:49.378 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:47:49.378 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-25T17:47:49.378 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T17:47:49.379 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-03-25T17:47:49.379 INFO:teuthology.orchestra.run.smithi192.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T17:47:49.379 INFO:teuthology.orchestra.run.smithi192.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T17:47:49.379 INFO:teuthology.orchestra.run.smithi192.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T17:47:49.379 INFO:teuthology.orchestra.run.smithi192.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T17:47:49.379 INFO:teuthology.orchestra.run.smithi192.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T17:47:49.379 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:47:49.379 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-25T17:47:49.379 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:47:49.379 INFO:teuthology.orchestra.run.smithi192.stdout:Install 6 Packages 2024-03-25T17:47:49.379 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:47:49.379 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 48 M 2024-03-25T17:47:49.379 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 204 M 2024-03-25T17:47:49.379 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-25T17:47:49.385 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T17:47:49.386 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:47:49.386 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-03-25T17:47:49.386 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:47:49.386 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-25T17:47:49.387 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T17:47:49.387 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-03-25T17:47:49.387 INFO:teuthology.orchestra.run.smithi006.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T17:47:49.387 INFO:teuthology.orchestra.run.smithi006.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T17:47:49.387 INFO:teuthology.orchestra.run.smithi006.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T17:47:49.387 INFO:teuthology.orchestra.run.smithi006.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T17:47:49.387 INFO:teuthology.orchestra.run.smithi006.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T17:47:49.387 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:47:49.387 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-25T17:47:49.387 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:47:49.387 INFO:teuthology.orchestra.run.smithi006.stdout:Install 6 Packages 2024-03-25T17:47:49.387 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:47:49.388 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 48 M 2024-03-25T17:47:49.388 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 204 M 2024-03-25T17:47:49.388 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-25T17:47:49.555 INFO:teuthology.orchestra.run.smithi192.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.5 MB/s | 217 kB 00:00 2024-03-25T17:47:49.622 INFO:teuthology.orchestra.run.smithi192.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 6.7 MB/s | 300 kB 00:00 2024-03-25T17:47:49.656 INFO:teuthology.orchestra.run.smithi192.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-03-25T17:47:49.844 INFO:teuthology.orchestra.run.smithi006.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-25T17:47:49.856 INFO:teuthology.orchestra.run.smithi192.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 936 kB/s | 187 kB 00:00 2024-03-25T17:47:49.894 INFO:teuthology.orchestra.run.smithi006.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-25T17:47:49.919 INFO:teuthology.orchestra.run.smithi006.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 63 kB 00:00 2024-03-25T17:47:50.398 INFO:teuthology.orchestra.run.smithi192.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 166 kB/s | 163 kB 00:00 2024-03-25T17:47:50.763 INFO:teuthology.orchestra.run.smithi006.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 222 kB/s | 187 kB 00:00 2024-03-25T17:47:51.339 INFO:teuthology.orchestra.run.smithi006.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 97 kB/s | 163 kB 00:01 2024-03-25T17:49:20.838 INFO:teuthology.orchestra.run.smithi192.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 529 kB/s | 47 MB 01:31 2024-03-25T17:49:20.839 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:49:20.840 INFO:teuthology.orchestra.run.smithi192.stdout:Total 539 kB/s | 48 MB 01:31 2024-03-25T17:49:20.852 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-25T17:49:20.873 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-25T17:49:20.873 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-25T17:49:21.259 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-25T17:49:21.259 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-25T17:49:21.851 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-25T17:49:21.928 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T17:49:21.972 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T17:49:22.011 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T17:49:22.059 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T17:49:22.081 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T17:49:26.655 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T17:49:26.814 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T17:49:29.081 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T17:49:29.081 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T17:49:29.081 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T17:49:29.081 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T17:49:29.081 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T17:49:29.081 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T17:49:29.476 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T17:49:29.476 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:49:29.476 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-25T17:49:29.476 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:49:29.476 INFO:teuthology.orchestra.run.smithi192.stdout: jq-1.6-16.el9.x86_64 2024-03-25T17:49:29.476 INFO:teuthology.orchestra.run.smithi192.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:49:29.476 INFO:teuthology.orchestra.run.smithi192.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T17:49:29.476 INFO:teuthology.orchestra.run.smithi192.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T17:49:29.476 INFO:teuthology.orchestra.run.smithi192.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T17:49:29.477 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:49:29.477 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T17:49:29.791 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph 2024-03-25T17:49:30.359 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:03:15 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T17:49:30.517 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T17:49:30.520 INFO:teuthology.orchestra.run.smithi192.stdout:========================================================================================= 2024-03-25T17:49:30.520 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-03-25T17:49:30.520 INFO:teuthology.orchestra.run.smithi192.stdout:========================================================================================= 2024-03-25T17:49:30.520 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-25T17:49:30.520 INFO:teuthology.orchestra.run.smithi192.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T17:49:30.520 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-03-25T17:49:30.521 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T17:49:30.521 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T17:49:30.521 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T17:49:30.521 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T17:49:30.521 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T17:49:30.521 INFO:teuthology.orchestra.run.smithi192.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T17:49:30.521 INFO:teuthology.orchestra.run.smithi192.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T17:49:30.521 INFO:teuthology.orchestra.run.smithi192.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T17:49:30.521 INFO:teuthology.orchestra.run.smithi192.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T17:49:30.521 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T17:49:30.521 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T17:49:30.521 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T17:49:30.521 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T17:49:30.522 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T17:49:30.522 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T17:49:30.522 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T17:49:30.522 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T17:49:30.522 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T17:49:30.522 INFO:teuthology.orchestra.run.smithi192.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T17:49:30.522 INFO:teuthology.orchestra.run.smithi192.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T17:49:30.522 INFO:teuthology.orchestra.run.smithi192.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T17:49:30.522 INFO:teuthology.orchestra.run.smithi192.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T17:49:30.522 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T17:49:30.522 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T17:49:30.522 INFO:teuthology.orchestra.run.smithi192.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T17:49:30.523 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T17:49:30.523 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T17:49:30.523 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T17:49:30.523 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T17:49:30.523 INFO:teuthology.orchestra.run.smithi192.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T17:49:30.523 INFO:teuthology.orchestra.run.smithi192.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T17:49:30.523 INFO:teuthology.orchestra.run.smithi192.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T17:49:30.523 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T17:49:30.523 INFO:teuthology.orchestra.run.smithi192.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T17:49:30.523 INFO:teuthology.orchestra.run.smithi192.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T17:49:30.523 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:49:30.523 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-25T17:49:30.523 INFO:teuthology.orchestra.run.smithi192.stdout:========================================================================================= 2024-03-25T17:49:30.523 INFO:teuthology.orchestra.run.smithi192.stdout:Install 36 Packages 2024-03-25T17:49:30.524 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:49:30.524 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 30 M 2024-03-25T17:49:30.524 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 107 M 2024-03-25T17:49:30.524 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-25T17:49:31.473 INFO:teuthology.orchestra.run.smithi192.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 15 kB/s | 6.6 kB 00:00 2024-03-25T17:49:35.848 INFO:teuthology.orchestra.run.smithi192.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 317 kB/s | 1.5 MB 00:04 2024-03-25T17:49:38.098 INFO:teuthology.orchestra.run.smithi192.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 308 kB/s | 2.1 MB 00:07 2024-03-25T17:49:39.556 INFO:teuthology.orchestra.run.smithi192.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 170 kB/s | 247 kB 00:01 2024-03-25T17:49:39.682 INFO:teuthology.orchestra.run.smithi192.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-25T17:49:39.723 INFO:teuthology.orchestra.run.smithi192.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T17:49:39.757 INFO:teuthology.orchestra.run.smithi192.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 951 kB/s | 32 kB 00:00 2024-03-25T17:49:39.790 INFO:teuthology.orchestra.run.smithi192.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T17:49:39.824 INFO:teuthology.orchestra.run.smithi192.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T17:49:39.866 INFO:teuthology.orchestra.run.smithi192.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-25T17:49:39.941 INFO:teuthology.orchestra.run.smithi192.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-25T17:49:39.975 INFO:teuthology.orchestra.run.smithi192.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T17:49:40.008 INFO:teuthology.orchestra.run.smithi192.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-25T17:49:40.042 INFO:teuthology.orchestra.run.smithi192.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-25T17:49:40.226 INFO:teuthology.orchestra.run.smithi192.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 681 kB/s | 125 kB 00:00 2024-03-25T17:49:40.309 INFO:teuthology.orchestra.run.smithi192.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 268 kB/s | 22 kB 00:00 2024-03-25T17:49:40.818 INFO:teuthology.orchestra.run.smithi192.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 297 kB/s | 151 kB 00:00 2024-03-25T17:49:40.918 INFO:teuthology.orchestra.run.smithi192.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 435 kB/s | 43 kB 00:00 2024-03-25T17:49:41.052 INFO:teuthology.orchestra.run.smithi192.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-03-25T17:49:41.244 INFO:teuthology.orchestra.run.smithi192.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.8 MB/s | 358 kB 00:00 2024-03-25T17:49:41.327 INFO:teuthology.orchestra.run.smithi192.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 128 kB/s | 11 kB 00:00 2024-03-25T17:49:41.420 INFO:teuthology.orchestra.run.smithi192.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 192 kB/s | 18 kB 00:00 2024-03-25T17:49:41.511 INFO:teuthology.orchestra.run.smithi192.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 253 kB/s | 23 kB 00:00 2024-03-25T17:49:41.603 INFO:teuthology.orchestra.run.smithi192.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 212 kB/s | 19 kB 00:00 2024-03-25T17:49:41.695 INFO:teuthology.orchestra.run.smithi192.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 215 kB/s | 20 kB 00:00 2024-03-25T17:49:41.787 INFO:teuthology.orchestra.run.smithi192.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 506 kB/s | 46 kB 00:00 2024-03-25T17:49:41.871 INFO:teuthology.orchestra.run.smithi192.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 946 kB/s | 79 kB 00:00 2024-03-25T17:49:41.988 INFO:teuthology.orchestra.run.smithi192.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-03-25T17:49:42.088 INFO:teuthology.orchestra.run.smithi192.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 164 kB/s | 16 kB 00:00 2024-03-25T17:49:42.188 INFO:teuthology.orchestra.run.smithi192.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 900 kB/s | 90 kB 00:00 2024-03-25T17:49:42.280 INFO:teuthology.orchestra.run.smithi192.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 390 kB/s | 36 kB 00:00 2024-03-25T17:49:42.389 INFO:teuthology.orchestra.run.smithi192.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-03-25T17:49:42.523 INFO:teuthology.orchestra.run.smithi192.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.1 MB/s | 427 kB 00:00 2024-03-25T17:49:42.623 INFO:teuthology.orchestra.run.smithi192.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 200 kB/s | 20 kB 00:00 2024-03-25T17:49:49.347 INFO:teuthology.orchestra.run.smithi192.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 268 kB/s | 4.7 MB 00:17 2024-03-25T17:50:10.275 INFO:teuthology.orchestra.run.smithi006.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 344 kB/s | 47 MB 02:20 2024-03-25T17:50:10.278 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:50:10.278 INFO:teuthology.orchestra.run.smithi006.stdout:Total 350 kB/s | 48 MB 02:20 2024-03-25T17:50:10.289 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-25T17:50:10.310 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-25T17:50:10.310 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-25T17:50:10.705 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-25T17:50:10.705 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-25T17:50:11.332 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-25T17:50:11.401 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T17:50:11.446 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T17:50:11.495 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T17:50:11.540 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T17:50:11.569 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T17:50:16.286 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T17:50:16.364 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T17:50:18.801 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T17:50:18.802 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T17:50:18.802 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T17:50:18.802 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T17:50:18.802 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T17:50:18.802 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T17:50:19.162 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T17:50:19.162 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:50:19.162 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-25T17:50:19.162 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:50:19.162 INFO:teuthology.orchestra.run.smithi006.stdout: jq-1.6-16.el9.x86_64 2024-03-25T17:50:19.162 INFO:teuthology.orchestra.run.smithi006.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:50:19.162 INFO:teuthology.orchestra.run.smithi006.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T17:50:19.162 INFO:teuthology.orchestra.run.smithi006.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T17:50:19.163 INFO:teuthology.orchestra.run.smithi006.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T17:50:19.163 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:50:19.163 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T17:50:19.461 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph 2024-03-25T17:50:20.033 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:04:02 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T17:50:20.192 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T17:50:20.194 INFO:teuthology.orchestra.run.smithi006.stdout:========================================================================================= 2024-03-25T17:50:20.194 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-03-25T17:50:20.194 INFO:teuthology.orchestra.run.smithi006.stdout:========================================================================================= 2024-03-25T17:50:20.195 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-25T17:50:20.195 INFO:teuthology.orchestra.run.smithi006.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T17:50:20.195 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-03-25T17:50:20.195 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T17:50:20.195 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T17:50:20.195 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T17:50:20.195 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T17:50:20.195 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T17:50:20.195 INFO:teuthology.orchestra.run.smithi006.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T17:50:20.195 INFO:teuthology.orchestra.run.smithi006.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T17:50:20.195 INFO:teuthology.orchestra.run.smithi006.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T17:50:20.195 INFO:teuthology.orchestra.run.smithi006.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T17:50:20.195 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T17:50:20.196 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T17:50:20.196 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T17:50:20.196 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T17:50:20.196 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T17:50:20.196 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T17:50:20.196 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T17:50:20.196 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T17:50:20.196 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T17:50:20.196 INFO:teuthology.orchestra.run.smithi006.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T17:50:20.196 INFO:teuthology.orchestra.run.smithi006.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T17:50:20.196 INFO:teuthology.orchestra.run.smithi006.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T17:50:20.196 INFO:teuthology.orchestra.run.smithi006.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T17:50:20.197 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T17:50:20.197 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T17:50:20.197 INFO:teuthology.orchestra.run.smithi006.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T17:50:20.197 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T17:50:20.197 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T17:50:20.197 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T17:50:20.197 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T17:50:20.197 INFO:teuthology.orchestra.run.smithi006.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T17:50:20.197 INFO:teuthology.orchestra.run.smithi006.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T17:50:20.197 INFO:teuthology.orchestra.run.smithi006.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T17:50:20.197 INFO:teuthology.orchestra.run.smithi006.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T17:50:20.197 INFO:teuthology.orchestra.run.smithi006.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T17:50:20.197 INFO:teuthology.orchestra.run.smithi006.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T17:50:20.197 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:50:20.198 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-25T17:50:20.198 INFO:teuthology.orchestra.run.smithi006.stdout:========================================================================================= 2024-03-25T17:50:20.198 INFO:teuthology.orchestra.run.smithi006.stdout:Install 36 Packages 2024-03-25T17:50:20.198 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:50:20.198 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 30 M 2024-03-25T17:50:20.198 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 107 M 2024-03-25T17:50:20.199 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-25T17:50:20.936 INFO:teuthology.orchestra.run.smithi006.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 18 kB/s | 6.6 kB 00:00 2024-03-25T17:50:23.551 INFO:teuthology.orchestra.run.smithi192.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 360 kB/s | 17 MB 00:47 2024-03-25T17:50:23.555 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:50:23.555 INFO:teuthology.orchestra.run.smithi192.stdout:Total 573 kB/s | 30 MB 00:53 2024-03-25T17:50:23.577 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-25T17:50:23.620 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-25T17:50:23.620 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-25T17:50:24.017 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-25T17:50:24.018 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-25T17:50:25.356 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-25T17:50:25.420 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T17:50:25.461 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T17:50:25.498 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T17:50:25.568 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T17:50:25.613 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T17:50:25.654 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T17:50:25.736 INFO:teuthology.orchestra.run.smithi006.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 421 kB/s | 2.1 MB 00:05 2024-03-25T17:50:25.752 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T17:50:25.865 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T17:50:25.907 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T17:50:25.978 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T17:50:26.041 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T17:50:26.079 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T17:50:26.128 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T17:50:26.191 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T17:50:26.245 INFO:teuthology.orchestra.run.smithi006.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 269 kB/s | 1.5 MB 00:05 2024-03-25T17:50:26.376 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T17:50:26.425 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T17:50:26.464 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T17:50:26.498 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T17:50:26.528 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T17:50:26.604 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T17:50:26.638 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T17:50:26.672 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T17:50:26.712 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T17:50:26.747 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T17:50:26.782 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T17:50:26.870 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T17:50:26.904 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T17:50:26.976 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T17:50:27.120 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T17:50:27.237 INFO:teuthology.orchestra.run.smithi006.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 249 kB/s | 247 kB 00:00 2024-03-25T17:50:27.238 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T17:50:27.371 INFO:teuthology.orchestra.run.smithi006.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-25T17:50:27.396 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T17:50:27.412 INFO:teuthology.orchestra.run.smithi006.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T17:50:27.426 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T17:50:27.451 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T17:50:27.451 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:50:27.451 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T17:50:27.451 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T17:50:27.451 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T17:50:27.451 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:50:27.463 INFO:teuthology.orchestra.run.smithi006.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 632 kB/s | 32 kB 00:00 2024-03-25T17:50:27.529 INFO:teuthology.orchestra.run.smithi006.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 710 kB/s | 47 kB 00:00 2024-03-25T17:50:27.571 INFO:teuthology.orchestra.run.smithi006.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 926 kB/s | 39 kB 00:00 2024-03-25T17:50:27.614 INFO:teuthology.orchestra.run.smithi006.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-25T17:50:27.697 INFO:teuthology.orchestra.run.smithi006.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-25T17:50:27.732 INFO:teuthology.orchestra.run.smithi006.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.9 MB/s | 103 kB 00:00 2024-03-25T17:50:27.773 INFO:teuthology.orchestra.run.smithi006.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.0 MB/s | 124 kB 00:00 2024-03-25T17:50:27.806 INFO:teuthology.orchestra.run.smithi006.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-25T17:50:27.990 INFO:teuthology.orchestra.run.smithi006.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 681 kB/s | 125 kB 00:00 2024-03-25T17:50:28.274 INFO:teuthology.orchestra.run.smithi006.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 79 kB/s | 22 kB 00:00 2024-03-25T17:50:28.524 INFO:teuthology.orchestra.run.smithi006.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 604 kB/s | 151 kB 00:00 2024-03-25T17:50:28.616 INFO:teuthology.orchestra.run.smithi006.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 474 kB/s | 43 kB 00:00 2024-03-25T17:50:28.699 INFO:teuthology.orchestra.run.smithi006.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-03-25T17:50:28.792 INFO:teuthology.orchestra.run.smithi006.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.8 MB/s | 358 kB 00:00 2024-03-25T17:50:28.858 INFO:teuthology.orchestra.run.smithi006.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 161 kB/s | 11 kB 00:00 2024-03-25T17:50:28.885 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T17:50:28.909 INFO:teuthology.orchestra.run.smithi006.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 353 kB/s | 18 kB 00:00 2024-03-25T17:50:28.911 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T17:50:28.911 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:50:28.911 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T17:50:28.911 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T17:50:28.911 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T17:50:28.911 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:50:28.976 INFO:teuthology.orchestra.run.smithi006.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 347 kB/s | 23 kB 00:00 2024-03-25T17:50:29.026 INFO:teuthology.orchestra.run.smithi006.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-03-25T17:50:29.068 INFO:teuthology.orchestra.run.smithi006.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 472 kB/s | 20 kB 00:00 2024-03-25T17:50:29.143 INFO:teuthology.orchestra.run.smithi006.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 618 kB/s | 46 kB 00:00 2024-03-25T17:50:29.218 INFO:teuthology.orchestra.run.smithi006.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-03-25T17:50:29.302 INFO:teuthology.orchestra.run.smithi006.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-03-25T17:50:29.361 INFO:teuthology.orchestra.run.smithi006.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 282 kB/s | 16 kB 00:00 2024-03-25T17:50:29.361 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T17:50:29.384 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T17:50:29.384 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:50:29.384 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T17:50:29.384 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T17:50:29.384 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T17:50:29.384 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:50:29.436 INFO:teuthology.orchestra.run.smithi006.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-03-25T17:50:29.503 INFO:teuthology.orchestra.run.smithi006.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 535 kB/s | 36 kB 00:00 2024-03-25T17:50:29.587 INFO:teuthology.orchestra.run.smithi006.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-25T17:50:29.595 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T17:50:29.617 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T17:50:29.617 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:50:29.617 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T17:50:29.617 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T17:50:29.617 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T17:50:29.618 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:50:29.684 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T17:50:29.701 INFO:teuthology.orchestra.run.smithi006.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.7 MB/s | 427 kB 00:00 2024-03-25T17:50:29.771 INFO:teuthology.orchestra.run.smithi006.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 287 kB/s | 20 kB 00:00 2024-03-25T17:50:31.337 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T17:50:31.337 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T17:50:31.338 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T17:50:31.338 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T17:50:31.338 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T17:50:31.338 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T17:50:31.338 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T17:50:31.338 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T17:50:31.338 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T17:50:31.338 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T17:50:31.338 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T17:50:31.338 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T17:50:31.338 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T17:50:31.338 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T17:50:31.338 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T17:50:31.338 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T17:50:31.339 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T17:50:31.339 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T17:50:31.339 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T17:50:31.339 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T17:50:31.339 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T17:50:31.339 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T17:50:31.339 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T17:50:31.339 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T17:50:31.339 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T17:50:31.339 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T17:50:31.339 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T17:50:31.339 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T17:50:31.339 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T17:50:31.339 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T17:50:31.339 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T17:50:31.339 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T17:50:31.340 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T17:50:31.340 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T17:50:31.340 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T17:50:31.340 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T17:50:32.221 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T17:50:32.222 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:50:32.222 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-25T17:50:32.222 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:50:32.222 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:50:32.222 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:50:32.222 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:50:32.222 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:50:32.222 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:50:32.222 INFO:teuthology.orchestra.run.smithi192.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T17:50:32.222 INFO:teuthology.orchestra.run.smithi192.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T17:50:32.222 INFO:teuthology.orchestra.run.smithi192.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T17:50:32.222 INFO:teuthology.orchestra.run.smithi192.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T17:50:32.223 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T17:50:32.223 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T17:50:32.223 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T17:50:32.223 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T17:50:32.223 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T17:50:32.223 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T17:50:32.223 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T17:50:32.223 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T17:50:32.223 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T17:50:32.223 INFO:teuthology.orchestra.run.smithi192.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T17:50:32.223 INFO:teuthology.orchestra.run.smithi192.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T17:50:32.223 INFO:teuthology.orchestra.run.smithi192.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T17:50:32.223 INFO:teuthology.orchestra.run.smithi192.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T17:50:32.223 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T17:50:32.224 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T17:50:32.224 INFO:teuthology.orchestra.run.smithi192.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T17:50:32.224 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T17:50:32.224 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T17:50:32.224 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T17:50:32.224 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T17:50:32.224 INFO:teuthology.orchestra.run.smithi192.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T17:50:32.224 INFO:teuthology.orchestra.run.smithi192.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T17:50:32.224 INFO:teuthology.orchestra.run.smithi192.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T17:50:32.224 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T17:50:32.224 INFO:teuthology.orchestra.run.smithi192.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T17:50:32.224 INFO:teuthology.orchestra.run.smithi192.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T17:50:32.224 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:50:32.224 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T17:50:32.589 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-base 2024-03-25T17:50:33.156 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:04:18 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T17:50:33.253 INFO:teuthology.orchestra.run.smithi192.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:50:33.304 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T17:50:33.305 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-03-25T17:50:33.306 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T17:50:33.364 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install cephadm 2024-03-25T17:50:33.924 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:04:18 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T17:50:34.075 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T17:50:34.075 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:50:34.075 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-03-25T17:50:34.075 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:50:34.075 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-25T17:50:34.075 INFO:teuthology.orchestra.run.smithi192.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T17:50:34.075 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:50:34.075 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-25T17:50:34.076 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:50:34.076 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-03-25T17:50:34.076 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:50:34.076 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 747 k 2024-03-25T17:50:34.076 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 754 k 2024-03-25T17:50:34.076 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-25T17:50:35.521 INFO:teuthology.orchestra.run.smithi006.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 329 kB/s | 4.7 MB 00:14 2024-03-25T17:50:36.868 INFO:teuthology.orchestra.run.smithi192.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 268 kB/s | 747 kB 00:02 2024-03-25T17:50:36.868 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:50:36.869 INFO:teuthology.orchestra.run.smithi192.stdout:Total 268 kB/s | 747 kB 00:02 2024-03-25T17:50:36.869 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-25T17:50:36.875 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-25T17:50:36.875 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-25T17:50:36.881 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-25T17:50:36.882 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-25T17:50:36.960 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-25T17:50:37.430 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T17:50:37.523 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T17:50:37.880 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T17:50:38.168 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T17:50:38.168 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:50:38.168 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-25T17:50:38.168 INFO:teuthology.orchestra.run.smithi192.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:50:38.168 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:50:38.168 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T17:50:38.399 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T17:50:38.967 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:04:23 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T17:50:39.121 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T17:50:39.122 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:50:39.122 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-03-25T17:50:39.122 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:50:39.123 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-25T17:50:39.123 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T17:50:39.123 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:50:39.123 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-25T17:50:39.123 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:50:39.123 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-03-25T17:50:39.123 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:50:39.123 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 145 k 2024-03-25T17:50:39.123 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 436 k 2024-03-25T17:50:39.123 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-25T17:50:39.966 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 173 kB/s | 145 kB 00:00 2024-03-25T17:50:39.966 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:50:39.966 INFO:teuthology.orchestra.run.smithi192.stdout:Total 172 kB/s | 145 kB 00:00 2024-03-25T17:50:39.966 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-25T17:50:39.976 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-25T17:50:39.976 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-25T17:50:40.032 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-25T17:50:40.033 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-25T17:50:40.172 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-25T17:50:40.241 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T17:50:40.266 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T17:50:40.266 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:50:40.266 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T17:50:40.266 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:50:41.283 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T17:50:41.283 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:50:41.283 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-25T17:50:41.283 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:50:41.283 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:50:41.283 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T17:50:41.539 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr 2024-03-25T17:50:42.107 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:04:27 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T17:50:42.205 INFO:teuthology.orchestra.run.smithi192.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:50:42.257 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T17:50:42.258 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-03-25T17:50:42.258 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T17:50:42.323 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T17:50:42.883 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:04:27 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T17:50:43.034 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T17:50:43.035 INFO:teuthology.orchestra.run.smithi192.stdout:====================================================================================== 2024-03-25T17:50:43.035 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-03-25T17:50:43.035 INFO:teuthology.orchestra.run.smithi192.stdout:====================================================================================== 2024-03-25T17:50:43.035 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-25T17:50:43.035 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T17:50:43.035 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-03-25T17:50:43.036 INFO:teuthology.orchestra.run.smithi192.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T17:50:43.036 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T17:50:43.036 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T17:50:43.036 INFO:teuthology.orchestra.run.smithi192.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T17:50:43.036 INFO:teuthology.orchestra.run.smithi192.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T17:50:43.036 INFO:teuthology.orchestra.run.smithi192.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T17:50:43.036 INFO:teuthology.orchestra.run.smithi192.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T17:50:43.036 INFO:teuthology.orchestra.run.smithi192.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T17:50:43.036 INFO:teuthology.orchestra.run.smithi192.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T17:50:43.036 INFO:teuthology.orchestra.run.smithi192.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T17:50:43.036 INFO:teuthology.orchestra.run.smithi192.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T17:50:43.036 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:50:43.036 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-25T17:50:43.037 INFO:teuthology.orchestra.run.smithi192.stdout:====================================================================================== 2024-03-25T17:50:43.037 INFO:teuthology.orchestra.run.smithi192.stdout:Install 12 Packages 2024-03-25T17:50:43.037 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:50:43.037 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 6.9 M 2024-03-25T17:50:43.037 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 26 M 2024-03-25T17:50:43.037 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-25T17:50:43.653 INFO:teuthology.orchestra.run.smithi192.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 62 kB/s | 31 kB 00:00 2024-03-25T17:50:43.669 INFO:teuthology.orchestra.run.smithi192.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 29 kB/s | 15 kB 00:00 2024-03-25T17:50:43.811 INFO:teuthology.orchestra.run.smithi192.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.0 MB/s | 238 kB 00:00 2024-03-25T17:50:43.853 INFO:teuthology.orchestra.run.smithi192.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.0 MB/s | 1.0 MB 00:00 2024-03-25T17:50:44.112 INFO:teuthology.orchestra.run.smithi192.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.8 MB/s | 863 kB 00:00 2024-03-25T17:50:44.146 INFO:teuthology.orchestra.run.smithi192.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.8 MB/s | 548 kB 00:00 2024-03-25T17:50:44.238 INFO:teuthology.orchestra.run.smithi192.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 155 kB/s | 19 kB 00:00 2024-03-25T17:50:44.296 INFO:teuthology.orchestra.run.smithi192.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 14 MB/s | 2.0 MB 00:00 2024-03-25T17:50:44.355 INFO:teuthology.orchestra.run.smithi192.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 527 kB/s | 31 kB 00:00 2024-03-25T17:50:44.414 INFO:teuthology.orchestra.run.smithi192.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 823 kB/s | 144 kB 00:00 2024-03-25T17:50:44.439 INFO:teuthology.orchestra.run.smithi192.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-03-25T17:50:47.805 INFO:teuthology.orchestra.run.smithi192.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 392 kB/s | 1.8 MB 00:04 2024-03-25T17:50:47.811 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:50:47.811 INFO:teuthology.orchestra.run.smithi192.stdout:Total 1.4 MB/s | 6.9 MB 00:04 2024-03-25T17:50:47.830 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-25T17:50:47.849 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-25T17:50:47.849 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-25T17:50:48.037 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-25T17:50:48.037 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-25T17:50:48.580 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-25T17:50:48.673 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T17:50:48.716 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T17:50:48.774 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T17:50:48.801 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T17:50:48.909 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T17:50:49.044 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T17:50:49.137 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T17:50:49.190 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T17:50:49.218 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T17:50:49.249 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T17:50:49.496 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T17:50:49.525 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T17:50:50.276 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T17:50:50.277 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T17:50:50.277 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T17:50:50.277 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T17:50:50.277 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T17:50:50.277 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T17:50:50.277 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T17:50:50.277 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T17:50:50.277 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T17:50:50.277 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T17:50:50.277 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T17:50:50.277 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T17:50:50.626 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T17:50:50.626 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:50:50.626 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-25T17:50:50.626 INFO:teuthology.orchestra.run.smithi192.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T17:50:50.626 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:50:50.626 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:50:50.626 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:50:50.626 INFO:teuthology.orchestra.run.smithi192.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T17:50:50.626 INFO:teuthology.orchestra.run.smithi192.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T17:50:50.626 INFO:teuthology.orchestra.run.smithi192.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T17:50:50.626 INFO:teuthology.orchestra.run.smithi192.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T17:50:50.626 INFO:teuthology.orchestra.run.smithi192.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T17:50:50.626 INFO:teuthology.orchestra.run.smithi192.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T17:50:50.626 INFO:teuthology.orchestra.run.smithi192.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T17:50:50.627 INFO:teuthology.orchestra.run.smithi192.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T17:50:50.627 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:50:50.627 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T17:50:50.896 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T17:50:51.457 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:04:36 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T17:50:51.608 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T17:50:51.609 INFO:teuthology.orchestra.run.smithi192.stdout:============================================================================================ 2024-03-25T17:50:51.609 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-03-25T17:50:51.609 INFO:teuthology.orchestra.run.smithi192.stdout:============================================================================================ 2024-03-25T17:50:51.609 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-25T17:50:51.609 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T17:50:51.609 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-03-25T17:50:51.609 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T17:50:51.609 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T17:50:51.610 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T17:50:51.610 INFO:teuthology.orchestra.run.smithi192.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T17:50:51.610 INFO:teuthology.orchestra.run.smithi192.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T17:50:51.610 INFO:teuthology.orchestra.run.smithi192.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T17:50:51.610 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T17:50:51.610 INFO:teuthology.orchestra.run.smithi192.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T17:50:51.610 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T17:50:51.610 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T17:50:51.610 INFO:teuthology.orchestra.run.smithi192.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T17:50:51.610 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:50:51.610 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-25T17:50:51.610 INFO:teuthology.orchestra.run.smithi192.stdout:============================================================================================ 2024-03-25T17:50:51.610 INFO:teuthology.orchestra.run.smithi192.stdout:Install 12 Packages 2024-03-25T17:50:51.611 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:50:51.611 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 37 M 2024-03-25T17:50:51.611 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 209 M 2024-03-25T17:50:51.611 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-25T17:50:51.912 INFO:teuthology.orchestra.run.smithi192.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 249 kB/s | 32 kB 00:00 2024-03-25T17:50:51.970 INFO:teuthology.orchestra.run.smithi192.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 495 kB/s | 17 kB 00:00 2024-03-25T17:50:52.004 INFO:teuthology.orchestra.run.smithi192.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-25T17:50:52.179 INFO:teuthology.orchestra.run.smithi192.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 27 MB/s | 4.7 MB 00:00 2024-03-25T17:50:52.329 INFO:teuthology.orchestra.run.smithi192.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2024-03-25T17:50:52.396 INFO:teuthology.orchestra.run.smithi192.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.9 MB/s | 3.0 MB 00:00 2024-03-25T17:50:52.430 INFO:teuthology.orchestra.run.smithi192.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.1 MB/s | 217 kB 00:00 2024-03-25T17:50:52.781 INFO:teuthology.orchestra.run.smithi192.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-03-25T17:50:52.906 INFO:teuthology.orchestra.run.smithi192.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 803 kB 00:00 2024-03-25T17:50:52.997 INFO:teuthology.orchestra.run.smithi192.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 902 kB/s | 195 kB 00:00 2024-03-25T17:50:53.123 INFO:teuthology.orchestra.run.smithi192.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-03-25T17:51:07.060 INFO:teuthology.orchestra.run.smithi006.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 416 kB/s | 17 MB 00:41 2024-03-25T17:51:07.063 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:51:07.063 INFO:teuthology.orchestra.run.smithi006.stdout:Total 648 kB/s | 30 MB 00:46 2024-03-25T17:51:07.085 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-25T17:51:07.130 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-25T17:51:07.130 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-25T17:51:07.563 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-25T17:51:07.563 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-25T17:51:08.888 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-25T17:51:08.962 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T17:51:09.011 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T17:51:09.048 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T17:51:09.121 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T17:51:09.172 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T17:51:09.205 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T17:51:09.300 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T17:51:09.386 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T17:51:09.441 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T17:51:09.515 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T17:51:09.568 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T17:51:09.605 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T17:51:09.654 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T17:51:09.710 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T17:51:09.903 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T17:51:09.959 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T17:51:10.014 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T17:51:10.049 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T17:51:10.087 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T17:51:10.166 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T17:51:10.197 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T17:51:10.231 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T17:51:10.271 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T17:51:10.307 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T17:51:10.349 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T17:51:10.442 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T17:51:10.480 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T17:51:10.563 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T17:51:10.704 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T17:51:10.828 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T17:51:10.997 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T17:51:11.018 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T17:51:11.043 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T17:51:11.043 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:51:11.043 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T17:51:11.043 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T17:51:11.043 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T17:51:11.044 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:51:12.578 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T17:51:12.605 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T17:51:12.605 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:51:12.605 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T17:51:12.605 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T17:51:12.605 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T17:51:12.605 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:51:13.070 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T17:51:13.093 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T17:51:13.093 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:51:13.093 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T17:51:13.093 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T17:51:13.093 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T17:51:13.093 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:51:13.313 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T17:51:13.335 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T17:51:13.335 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:51:13.335 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T17:51:13.335 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T17:51:13.335 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T17:51:13.335 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:51:13.362 INFO:teuthology.orchestra.run.smithi192.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 351 kB/s | 7.4 MB 00:21 2024-03-25T17:51:13.367 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:51:13.367 INFO:teuthology.orchestra.run.smithi192.stdout:Total 1.7 MB/s | 37 MB 00:21 2024-03-25T17:51:13.404 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T17:51:13.410 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-25T17:51:13.437 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-25T17:51:13.437 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-25T17:51:13.791 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-25T17:51:13.791 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-25T17:51:14.535 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-25T17:51:14.618 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T17:51:14.650 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T17:51:14.743 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T17:51:14.786 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T17:51:14.963 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T17:51:15.002 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T17:51:15.089 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T17:51:15.114 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T17:51:15.114 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T17:51:15.114 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T17:51:15.114 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T17:51:15.114 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T17:51:15.114 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T17:51:15.114 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T17:51:15.114 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T17:51:15.114 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T17:51:15.114 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T17:51:15.115 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T17:51:15.115 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T17:51:15.115 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T17:51:15.115 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T17:51:15.115 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T17:51:15.115 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T17:51:15.115 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T17:51:15.115 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T17:51:15.115 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T17:51:15.115 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T17:51:15.116 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T17:51:15.116 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T17:51:15.116 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T17:51:15.116 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T17:51:15.116 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T17:51:15.116 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T17:51:15.116 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T17:51:15.116 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T17:51:15.116 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T17:51:15.116 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T17:51:15.116 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T17:51:15.116 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T17:51:15.117 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T17:51:15.117 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T17:51:15.117 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T17:51:15.117 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T17:51:15.735 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T17:51:15.847 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T17:51:16.258 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T17:51:16.258 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:51:16.258 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-25T17:51:16.258 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:16.258 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:16.258 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:16.258 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:51:16.258 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:16.258 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:16.258 INFO:teuthology.orchestra.run.smithi006.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T17:51:16.258 INFO:teuthology.orchestra.run.smithi006.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T17:51:16.259 INFO:teuthology.orchestra.run.smithi006.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T17:51:16.259 INFO:teuthology.orchestra.run.smithi006.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T17:51:16.259 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T17:51:16.259 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T17:51:16.259 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T17:51:16.259 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T17:51:16.259 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T17:51:16.259 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T17:51:16.259 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T17:51:16.259 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T17:51:16.259 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T17:51:16.259 INFO:teuthology.orchestra.run.smithi006.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T17:51:16.259 INFO:teuthology.orchestra.run.smithi006.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T17:51:16.259 INFO:teuthology.orchestra.run.smithi006.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T17:51:16.260 INFO:teuthology.orchestra.run.smithi006.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T17:51:16.260 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T17:51:16.260 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T17:51:16.260 INFO:teuthology.orchestra.run.smithi006.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T17:51:16.260 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T17:51:16.260 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T17:51:16.260 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T17:51:16.260 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T17:51:16.260 INFO:teuthology.orchestra.run.smithi006.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T17:51:16.260 INFO:teuthology.orchestra.run.smithi006.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T17:51:16.260 INFO:teuthology.orchestra.run.smithi006.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T17:51:16.260 INFO:teuthology.orchestra.run.smithi006.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T17:51:16.260 INFO:teuthology.orchestra.run.smithi006.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T17:51:16.260 INFO:teuthology.orchestra.run.smithi006.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T17:51:16.260 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:51:16.261 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T17:51:16.616 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-base 2024-03-25T17:51:17.085 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T17:51:17.199 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:04:59 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T17:51:17.298 INFO:teuthology.orchestra.run.smithi006.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:51:17.353 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T17:51:17.354 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-03-25T17:51:17.354 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T17:51:17.414 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install cephadm 2024-03-25T17:51:17.985 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:04:59 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T17:51:18.111 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T17:51:18.136 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T17:51:18.140 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T17:51:18.141 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:51:18.141 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-03-25T17:51:18.141 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:51:18.141 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-25T17:51:18.141 INFO:teuthology.orchestra.run.smithi006.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T17:51:18.141 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:51:18.141 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-25T17:51:18.141 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:51:18.141 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-03-25T17:51:18.141 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:51:18.142 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 747 k 2024-03-25T17:51:18.142 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 754 k 2024-03-25T17:51:18.142 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-25T17:51:20.328 INFO:teuthology.orchestra.run.smithi006.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 342 kB/s | 747 kB 00:02 2024-03-25T17:51:20.329 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:51:20.329 INFO:teuthology.orchestra.run.smithi006.stdout:Total 342 kB/s | 747 kB 00:02 2024-03-25T17:51:20.330 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-25T17:51:20.336 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-25T17:51:20.336 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-25T17:51:20.342 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-25T17:51:20.343 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-25T17:51:20.441 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-25T17:51:20.603 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T17:51:20.603 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T17:51:20.603 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T17:51:20.603 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T17:51:20.603 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T17:51:20.603 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T17:51:20.603 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T17:51:20.603 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T17:51:20.603 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T17:51:20.603 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T17:51:20.604 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T17:51:20.604 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T17:51:20.900 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T17:51:21.019 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T17:51:21.081 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T17:51:21.082 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:51:21.082 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-25T17:51:21.082 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:51:21.082 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T17:51:21.082 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T17:51:21.082 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T17:51:21.082 INFO:teuthology.orchestra.run.smithi192.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T17:51:21.082 INFO:teuthology.orchestra.run.smithi192.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T17:51:21.082 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T17:51:21.082 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T17:51:21.082 INFO:teuthology.orchestra.run.smithi192.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T17:51:21.082 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T17:51:21.082 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T17:51:21.083 INFO:teuthology.orchestra.run.smithi192.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T17:51:21.083 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:51:21.083 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T17:51:21.372 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T17:51:21.495 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr-rook 2024-03-25T17:51:21.798 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T17:51:21.798 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:51:21.798 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-25T17:51:21.798 INFO:teuthology.orchestra.run.smithi006.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:51:21.798 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:51:21.798 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T17:51:22.016 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T17:51:22.065 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:05:07 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T17:51:22.214 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T17:51:22.215 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================== 2024-03-25T17:51:22.215 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-03-25T17:51:22.215 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================== 2024-03-25T17:51:22.215 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-25T17:51:22.215 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T17:51:22.216 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-03-25T17:51:22.216 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T17:51:22.216 INFO:teuthology.orchestra.run.smithi192.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T17:51:22.216 INFO:teuthology.orchestra.run.smithi192.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T17:51:22.216 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T17:51:22.216 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T17:51:22.216 INFO:teuthology.orchestra.run.smithi192.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T17:51:22.216 INFO:teuthology.orchestra.run.smithi192.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T17:51:22.216 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T17:51:22.216 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T17:51:22.216 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T17:51:22.216 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T17:51:22.216 INFO:teuthology.orchestra.run.smithi192.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T17:51:22.216 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:51:22.217 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-25T17:51:22.217 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================== 2024-03-25T17:51:22.217 INFO:teuthology.orchestra.run.smithi192.stdout:Install 13 Packages 2024-03-25T17:51:22.217 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:51:22.217 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 2.0 M 2024-03-25T17:51:22.217 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 26 M 2024-03-25T17:51:22.217 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-25T17:51:22.355 INFO:teuthology.orchestra.run.smithi192.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 259 kB/s | 24 kB 00:00 2024-03-25T17:51:22.397 INFO:teuthology.orchestra.run.smithi192.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 133 kB/s | 18 kB 00:00 2024-03-25T17:51:22.456 INFO:teuthology.orchestra.run.smithi192.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 174 kB 00:00 2024-03-25T17:51:22.481 INFO:teuthology.orchestra.run.smithi192.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-25T17:51:22.507 INFO:teuthology.orchestra.run.smithi192.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T17:51:22.524 INFO:teuthology.orchestra.run.smithi192.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-25T17:51:22.592 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:05:04 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T17:51:22.615 INFO:teuthology.orchestra.run.smithi192.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 164 kB/s | 15 kB 00:00 2024-03-25T17:51:22.632 INFO:teuthology.orchestra.run.smithi192.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 257 kB/s | 32 kB 00:00 2024-03-25T17:51:22.748 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T17:51:22.748 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:51:22.749 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-03-25T17:51:22.749 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:51:22.749 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-25T17:51:22.749 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T17:51:22.749 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:51:22.749 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-25T17:51:22.749 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:51:22.749 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-03-25T17:51:22.749 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:51:22.749 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 145 k 2024-03-25T17:51:22.749 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 436 k 2024-03-25T17:51:22.749 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-25T17:51:22.799 INFO:teuthology.orchestra.run.smithi192.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 1.1 MB/s | 203 kB 00:00 2024-03-25T17:51:22.877 INFO:teuthology.orchestra.run.smithi192.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 787 kB/s | 59 kB 00:00 2024-03-25T17:51:22.975 INFO:teuthology.orchestra.run.smithi192.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 896 kB/s | 90 kB 00:00 2024-03-25T17:51:22.992 INFO:teuthology.orchestra.run.smithi192.stdout:(12/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el 71 kB/s | 52 kB 00:00 2024-03-25T17:51:23.184 INFO:teuthology.orchestra.run.smithi192.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.9 MB/s | 1.0 MB 00:00 2024-03-25T17:51:23.186 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:51:23.186 INFO:teuthology.orchestra.run.smithi192.stdout:Total 2.1 MB/s | 2.0 MB 00:00 2024-03-25T17:51:23.187 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-25T17:51:23.202 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-25T17:51:23.202 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-25T17:51:23.301 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-25T17:51:23.302 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-25T17:51:23.693 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 154 kB/s | 145 kB 00:00 2024-03-25T17:51:23.694 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:51:23.694 INFO:teuthology.orchestra.run.smithi006.stdout:Total 154 kB/s | 145 kB 00:00 2024-03-25T17:51:23.694 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-25T17:51:23.704 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-25T17:51:23.704 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-25T17:51:23.763 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-25T17:51:23.763 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-25T17:51:23.815 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-25T17:51:23.908 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-25T17:51:23.969 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T17:51:23.978 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T17:51:23.993 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T17:51:23.993 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:51:23.994 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T17:51:23.994 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:51:24.026 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T17:51:24.081 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T17:51:24.128 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T17:51:24.169 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T17:51:24.262 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T17:51:24.370 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T17:51:24.428 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T17:51:25.026 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T17:51:25.026 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:51:25.027 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-25T17:51:25.027 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:25.027 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:51:25.027 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T17:51:25.148 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T17:51:25.201 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T17:51:25.243 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T17:51:25.292 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr 2024-03-25T17:51:25.299 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T17:51:25.321 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T17:51:25.866 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:05:07 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T17:51:25.967 INFO:teuthology.orchestra.run.smithi006.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:51:26.024 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T17:51:26.025 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-03-25T17:51:26.025 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T17:51:26.087 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T17:51:26.095 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T17:51:26.095 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T17:51:26.095 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T17:51:26.095 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T17:51:26.095 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T17:51:26.095 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T17:51:26.095 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T17:51:26.095 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T17:51:26.095 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T17:51:26.095 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T17:51:26.096 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T17:51:26.096 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T17:51:26.096 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T17:51:26.454 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T17:51:26.454 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:51:26.454 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-25T17:51:26.454 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:51:26.454 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T17:51:26.454 INFO:teuthology.orchestra.run.smithi192.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T17:51:26.454 INFO:teuthology.orchestra.run.smithi192.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T17:51:26.454 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T17:51:26.454 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T17:51:26.454 INFO:teuthology.orchestra.run.smithi192.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T17:51:26.454 INFO:teuthology.orchestra.run.smithi192.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T17:51:26.454 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T17:51:26.455 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T17:51:26.455 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T17:51:26.455 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T17:51:26.455 INFO:teuthology.orchestra.run.smithi192.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T17:51:26.455 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:51:26.455 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T17:51:26.653 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:05:08 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T17:51:26.732 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T17:51:26.810 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T17:51:26.811 INFO:teuthology.orchestra.run.smithi006.stdout:====================================================================================== 2024-03-25T17:51:26.811 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-03-25T17:51:26.811 INFO:teuthology.orchestra.run.smithi006.stdout:====================================================================================== 2024-03-25T17:51:26.812 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-25T17:51:26.812 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T17:51:26.812 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-03-25T17:51:26.812 INFO:teuthology.orchestra.run.smithi006.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T17:51:26.812 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T17:51:26.812 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T17:51:26.812 INFO:teuthology.orchestra.run.smithi006.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T17:51:26.812 INFO:teuthology.orchestra.run.smithi006.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T17:51:26.812 INFO:teuthology.orchestra.run.smithi006.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T17:51:26.812 INFO:teuthology.orchestra.run.smithi006.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T17:51:26.812 INFO:teuthology.orchestra.run.smithi006.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T17:51:26.812 INFO:teuthology.orchestra.run.smithi006.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T17:51:26.812 INFO:teuthology.orchestra.run.smithi006.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T17:51:26.812 INFO:teuthology.orchestra.run.smithi006.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T17:51:26.813 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:51:26.813 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-25T17:51:26.813 INFO:teuthology.orchestra.run.smithi006.stdout:====================================================================================== 2024-03-25T17:51:26.813 INFO:teuthology.orchestra.run.smithi006.stdout:Install 12 Packages 2024-03-25T17:51:26.813 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:51:26.813 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 6.9 M 2024-03-25T17:51:26.813 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 26 M 2024-03-25T17:51:26.814 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-25T17:51:27.292 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:05:12 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T17:51:27.442 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T17:51:27.442 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================== 2024-03-25T17:51:27.443 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-03-25T17:51:27.443 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================== 2024-03-25T17:51:27.443 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-25T17:51:27.443 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T17:51:27.443 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-03-25T17:51:27.443 INFO:teuthology.orchestra.run.smithi192.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T17:51:27.443 INFO:teuthology.orchestra.run.smithi192.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T17:51:27.443 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T17:51:27.443 INFO:teuthology.orchestra.run.smithi192.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T17:51:27.443 INFO:teuthology.orchestra.run.smithi192.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T17:51:27.443 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:51:27.443 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-25T17:51:27.443 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================== 2024-03-25T17:51:27.443 INFO:teuthology.orchestra.run.smithi192.stdout:Install 6 Packages 2024-03-25T17:51:27.443 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:51:27.444 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 6.8 M 2024-03-25T17:51:27.444 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 33 M 2024-03-25T17:51:27.444 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-25T17:51:27.530 INFO:teuthology.orchestra.run.smithi006.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 64 kB/s | 31 kB 00:00 2024-03-25T17:51:27.564 INFO:teuthology.orchestra.run.smithi006.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 29 kB/s | 15 kB 00:00 2024-03-25T17:51:27.714 INFO:teuthology.orchestra.run.smithi006.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.9 MB/s | 238 kB 00:00 2024-03-25T17:51:27.756 INFO:teuthology.orchestra.run.smithi006.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.5 MB/s | 1.0 MB 00:00 2024-03-25T17:51:27.893 INFO:teuthology.orchestra.run.smithi192.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-25T17:51:27.923 INFO:teuthology.orchestra.run.smithi006.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.1 MB/s | 863 kB 00:00 2024-03-25T17:51:27.927 INFO:teuthology.orchestra.run.smithi192.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 850 kB/s | 248 kB 00:00 2024-03-25T17:51:28.035 INFO:teuthology.orchestra.run.smithi192.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 534 kB/s | 58 kB 00:00 2024-03-25T17:51:28.069 INFO:teuthology.orchestra.run.smithi192.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 3.1 MB/s | 548 kB 00:00 2024-03-25T17:51:28.094 INFO:teuthology.orchestra.run.smithi192.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 873 kB/s | 51 kB 00:00 2024-03-25T17:51:28.207 INFO:teuthology.orchestra.run.smithi006.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 69 kB/s | 19 kB 00:00 2024-03-25T17:51:28.424 INFO:teuthology.orchestra.run.smithi006.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 821 kB/s | 548 kB 00:00 2024-03-25T17:51:28.541 INFO:teuthology.orchestra.run.smithi006.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.2 MB/s | 144 kB 00:00 2024-03-25T17:51:28.649 INFO:teuthology.orchestra.run.smithi006.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 284 kB/s | 31 kB 00:00 2024-03-25T17:51:28.799 INFO:teuthology.orchestra.run.smithi006.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.2 MB/s | 188 kB 00:00 2024-03-25T17:51:28.811 INFO:teuthology.orchestra.run.smithi192.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 117 kB/s | 138 kB 00:01 2024-03-25T17:51:28.813 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:51:28.813 INFO:teuthology.orchestra.run.smithi192.stdout:Total 5.0 MB/s | 6.8 MB 00:01 2024-03-25T17:51:28.827 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-25T17:51:28.839 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-25T17:51:28.839 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-25T17:51:28.866 INFO:teuthology.orchestra.run.smithi006.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.1 MB/s | 2.0 MB 00:00 2024-03-25T17:51:28.932 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-25T17:51:28.933 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-25T17:51:29.241 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-25T17:51:29.410 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T17:51:29.463 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T17:51:29.994 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T17:51:30.074 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T17:51:30.135 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T17:51:30.166 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T17:51:30.931 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T17:51:30.931 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T17:51:30.931 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T17:51:30.931 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T17:51:30.931 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T17:51:30.931 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T17:51:31.244 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T17:51:31.244 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:51:31.244 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-25T17:51:31.244 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:51:31.244 INFO:teuthology.orchestra.run.smithi192.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T17:51:31.244 INFO:teuthology.orchestra.run.smithi192.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T17:51:31.244 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T17:51:31.244 INFO:teuthology.orchestra.run.smithi192.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T17:51:31.244 INFO:teuthology.orchestra.run.smithi192.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T17:51:31.245 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:51:31.245 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T17:51:31.493 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-fuse 2024-03-25T17:51:32.055 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:05:17 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T17:51:32.204 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T17:51:32.205 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:51:32.205 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-03-25T17:51:32.205 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:51:32.205 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-25T17:51:32.205 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T17:51:32.205 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-03-25T17:51:32.205 INFO:teuthology.orchestra.run.smithi192.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T17:51:32.205 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:51:32.205 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-25T17:51:32.205 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:51:32.205 INFO:teuthology.orchestra.run.smithi192.stdout:Install 2 Packages 2024-03-25T17:51:32.206 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:51:32.206 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 964 k 2024-03-25T17:51:32.206 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 2.8 M 2024-03-25T17:51:32.206 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-25T17:51:32.313 INFO:teuthology.orchestra.run.smithi192.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 737 kB/s | 79 kB 00:00 2024-03-25T17:51:33.416 INFO:teuthology.orchestra.run.smithi006.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 286 kB/s | 1.8 MB 00:06 2024-03-25T17:51:33.420 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:51:33.421 INFO:teuthology.orchestra.run.smithi006.stdout:Total 1.0 MB/s | 6.9 MB 00:06 2024-03-25T17:51:33.441 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-25T17:51:33.462 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-25T17:51:33.462 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-25T17:51:33.640 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-25T17:51:33.641 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-25T17:51:34.276 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-25T17:51:34.359 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T17:51:34.394 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T17:51:34.442 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T17:51:34.471 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T17:51:34.588 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T17:51:34.717 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T17:51:34.811 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T17:51:34.868 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T17:51:34.895 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T17:51:34.928 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T17:51:35.183 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T17:51:35.219 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T17:51:36.029 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T17:51:36.029 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T17:51:36.029 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T17:51:36.029 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T17:51:36.029 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T17:51:36.029 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T17:51:36.029 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T17:51:36.029 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T17:51:36.029 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T17:51:36.030 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T17:51:36.030 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T17:51:36.030 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T17:51:36.377 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T17:51:36.378 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:51:36.378 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-25T17:51:36.378 INFO:teuthology.orchestra.run.smithi006.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T17:51:36.378 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:51:36.378 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:51:36.378 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:51:36.378 INFO:teuthology.orchestra.run.smithi006.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T17:51:36.378 INFO:teuthology.orchestra.run.smithi006.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T17:51:36.378 INFO:teuthology.orchestra.run.smithi006.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T17:51:36.378 INFO:teuthology.orchestra.run.smithi006.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T17:51:36.378 INFO:teuthology.orchestra.run.smithi006.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T17:51:36.378 INFO:teuthology.orchestra.run.smithi006.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T17:51:36.378 INFO:teuthology.orchestra.run.smithi006.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T17:51:36.379 INFO:teuthology.orchestra.run.smithi006.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T17:51:36.379 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:51:36.379 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T17:51:36.638 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T17:51:36.721 INFO:teuthology.orchestra.run.smithi192.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 196 kB/s | 885 kB 00:04 2024-03-25T17:51:36.723 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:51:36.723 INFO:teuthology.orchestra.run.smithi192.stdout:Total 213 kB/s | 964 kB 00:04 2024-03-25T17:51:36.724 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-25T17:51:36.747 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-25T17:51:36.747 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-25T17:51:36.825 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-25T17:51:36.826 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-25T17:51:37.004 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-25T17:51:37.156 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T17:51:37.208 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:05:19 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T17:51:37.221 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T17:51:37.365 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T17:51:37.366 INFO:teuthology.orchestra.run.smithi006.stdout:============================================================================================ 2024-03-25T17:51:37.367 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-03-25T17:51:37.367 INFO:teuthology.orchestra.run.smithi006.stdout:============================================================================================ 2024-03-25T17:51:37.367 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-25T17:51:37.367 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T17:51:37.367 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-03-25T17:51:37.367 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T17:51:37.367 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T17:51:37.367 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T17:51:37.367 INFO:teuthology.orchestra.run.smithi006.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T17:51:37.367 INFO:teuthology.orchestra.run.smithi006.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T17:51:37.367 INFO:teuthology.orchestra.run.smithi006.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T17:51:37.367 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T17:51:37.368 INFO:teuthology.orchestra.run.smithi006.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T17:51:37.368 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T17:51:37.368 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T17:51:37.368 INFO:teuthology.orchestra.run.smithi006.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T17:51:37.368 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:51:37.368 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-25T17:51:37.368 INFO:teuthology.orchestra.run.smithi006.stdout:============================================================================================ 2024-03-25T17:51:37.368 INFO:teuthology.orchestra.run.smithi006.stdout:Install 12 Packages 2024-03-25T17:51:37.368 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:51:37.369 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 37 M 2024-03-25T17:51:37.369 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 209 M 2024-03-25T17:51:37.369 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-25T17:51:37.664 INFO:teuthology.orchestra.run.smithi006.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 271 kB/s | 32 kB 00:00 2024-03-25T17:51:37.723 INFO:teuthology.orchestra.run.smithi006.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 502 kB/s | 17 kB 00:00 2024-03-25T17:51:37.773 INFO:teuthology.orchestra.run.smithi006.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 684 kB/s | 34 kB 00:00 2024-03-25T17:51:37.909 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T17:51:37.909 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T17:51:37.965 INFO:teuthology.orchestra.run.smithi006.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 25 MB/s | 4.7 MB 00:00 2024-03-25T17:51:38.115 INFO:teuthology.orchestra.run.smithi006.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2024-03-25T17:51:38.165 INFO:teuthology.orchestra.run.smithi006.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 4.3 MB/s | 217 kB 00:00 2024-03-25T17:51:38.200 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T17:51:38.201 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:51:38.201 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-25T17:51:38.201 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T17:51:38.201 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:51:38.201 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T17:51:38.283 INFO:teuthology.orchestra.run.smithi006.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.1 MB/s | 3.0 MB 00:00 2024-03-25T17:51:38.497 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-volume 2024-03-25T17:51:38.541 INFO:teuthology.orchestra.run.smithi006.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 42 MB/s | 16 MB 00:00 2024-03-25T17:51:38.742 INFO:teuthology.orchestra.run.smithi006.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 803 kB 00:00 2024-03-25T17:51:38.775 INFO:teuthology.orchestra.run.smithi006.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 837 kB/s | 195 kB 00:00 2024-03-25T17:51:38.925 INFO:teuthology.orchestra.run.smithi006.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.3 MB/s | 245 kB 00:00 2024-03-25T17:51:39.070 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:05:24 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T17:51:39.228 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T17:51:39.228 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:51:39.228 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-03-25T17:51:39.228 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:51:39.229 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-25T17:51:39.229 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T17:51:39.229 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-03-25T17:51:39.229 INFO:teuthology.orchestra.run.smithi192.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T17:51:39.229 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:51:39.229 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-25T17:51:39.229 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:51:39.229 INFO:teuthology.orchestra.run.smithi192.stdout:Install 2 Packages 2024-03-25T17:51:39.229 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:51:39.229 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 332 k 2024-03-25T17:51:39.229 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 1.5 M 2024-03-25T17:51:39.229 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-25T17:51:39.367 INFO:teuthology.orchestra.run.smithi192.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 500 kB/s | 69 kB 00:00 2024-03-25T17:51:41.009 INFO:teuthology.orchestra.run.smithi192.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 148 kB/s | 264 kB 00:01 2024-03-25T17:51:41.010 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:51:41.011 INFO:teuthology.orchestra.run.smithi192.stdout:Total 186 kB/s | 332 kB 00:01 2024-03-25T17:51:41.011 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-25T17:51:41.021 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-25T17:51:41.021 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-25T17:51:41.062 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-25T17:51:41.062 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-25T17:51:41.217 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-25T17:51:41.402 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T17:51:41.453 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T17:51:41.469 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T17:51:41.469 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:51:41.469 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T17:51:41.469 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:51:42.277 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T17:51:42.608 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T17:51:42.608 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:51:42.608 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-25T17:51:42.608 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:51:42.608 INFO:teuthology.orchestra.run.smithi192.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T17:51:42.608 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:51:42.608 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T17:51:42.957 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install librados-devel 2024-03-25T17:51:43.539 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:05:28 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T17:51:43.689 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T17:51:43.689 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:51:43.689 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-03-25T17:51:43.689 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:51:43.690 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-25T17:51:43.690 INFO:teuthology.orchestra.run.smithi192.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T17:51:43.690 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:51:43.690 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-25T17:51:43.690 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:51:43.690 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-03-25T17:51:43.690 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:51:43.690 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 127 k 2024-03-25T17:51:43.690 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 456 k 2024-03-25T17:51:43.690 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-25T17:51:44.573 INFO:teuthology.orchestra.run.smithi192.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 143 kB/s | 127 kB 00:00 2024-03-25T17:51:44.574 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:51:44.574 INFO:teuthology.orchestra.run.smithi192.stdout:Total 143 kB/s | 127 kB 00:00 2024-03-25T17:51:44.574 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-25T17:51:44.580 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-25T17:51:44.580 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-25T17:51:44.637 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-25T17:51:44.637 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-25T17:51:44.785 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-25T17:51:44.895 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:51:45.368 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:51:45.674 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:51:45.674 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:51:45.674 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-25T17:51:45.674 INFO:teuthology.orchestra.run.smithi192.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:45.674 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:51:45.674 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T17:51:45.910 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install libcephfs2 2024-03-25T17:51:46.486 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:05:31 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T17:51:46.587 INFO:teuthology.orchestra.run.smithi192.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:51:46.642 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T17:51:46.643 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-03-25T17:51:46.643 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T17:51:46.716 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install libcephfs-devel 2024-03-25T17:51:47.279 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:05:32 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T17:51:47.429 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T17:51:47.430 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:51:47.430 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-03-25T17:51:47.430 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:51:47.430 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-25T17:51:47.430 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T17:51:47.430 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:51:47.430 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-25T17:51:47.430 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:51:47.430 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-03-25T17:51:47.430 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:51:47.431 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 31 k 2024-03-25T17:51:47.431 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 140 k 2024-03-25T17:51:47.431 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-25T17:51:47.893 INFO:teuthology.orchestra.run.smithi192.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 67 kB/s | 31 kB 00:00 2024-03-25T17:51:47.893 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:51:47.894 INFO:teuthology.orchestra.run.smithi192.stdout:Total 67 kB/s | 31 kB 00:00 2024-03-25T17:51:47.894 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-25T17:51:47.897 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-25T17:51:47.897 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-25T17:51:47.933 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-25T17:51:47.934 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-25T17:51:48.055 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-25T17:51:48.150 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:51:48.543 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:51:48.806 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:51:48.807 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:51:48.807 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-25T17:51:48.807 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:48.807 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:51:48.807 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T17:51:48.961 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install librados2 2024-03-25T17:51:49.543 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:05:34 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T17:51:49.650 INFO:teuthology.orchestra.run.smithi192.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:51:49.706 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T17:51:49.707 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-03-25T17:51:49.707 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T17:51:49.766 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install librbd1 2024-03-25T17:51:50.357 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:05:35 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T17:51:50.458 INFO:teuthology.orchestra.run.smithi192.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:51:50.512 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T17:51:50.513 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-03-25T17:51:50.513 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T17:51:50.563 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-rados 2024-03-25T17:51:51.125 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:05:36 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T17:51:51.221 INFO:teuthology.orchestra.run.smithi192.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:51:51.272 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T17:51:51.273 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-03-25T17:51:51.274 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T17:51:51.328 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-rgw 2024-03-25T17:51:51.887 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:05:36 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T17:51:51.983 INFO:teuthology.orchestra.run.smithi192.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:51:52.035 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T17:51:52.036 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-03-25T17:51:52.036 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T17:51:52.104 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-cephfs 2024-03-25T17:51:52.667 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:05:37 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T17:51:52.763 INFO:teuthology.orchestra.run.smithi192.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:51:52.815 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T17:51:52.816 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-03-25T17:51:52.816 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T17:51:52.869 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-rbd 2024-03-25T17:51:53.427 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:05:38 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T17:51:53.523 INFO:teuthology.orchestra.run.smithi192.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:51:53.575 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T17:51:53.576 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-03-25T17:51:53.577 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T17:51:53.631 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install rbd-fuse 2024-03-25T17:51:54.194 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:05:39 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T17:51:54.343 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T17:51:54.344 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:51:54.344 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2024-03-25T17:51:54.344 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:51:54.344 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-25T17:51:54.344 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T17:51:54.344 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:51:54.344 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-25T17:51:54.344 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:51:54.344 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-03-25T17:51:54.345 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:51:54.345 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 85 k 2024-03-25T17:51:54.345 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 231 k 2024-03-25T17:51:54.345 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-25T17:51:54.408 INFO:teuthology.orchestra.run.smithi006.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 449 kB/s | 7.4 MB 00:16 2024-03-25T17:51:54.410 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:51:54.410 INFO:teuthology.orchestra.run.smithi006.stdout:Total 2.2 MB/s | 37 MB 00:17 2024-03-25T17:51:54.455 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-25T17:51:54.482 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-25T17:51:54.483 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-25T17:51:54.863 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-25T17:51:54.863 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-25T17:51:54.922 INFO:teuthology.orchestra.run.smithi192.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 148 kB/s | 85 kB 00:00 2024-03-25T17:51:54.922 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:51:54.922 INFO:teuthology.orchestra.run.smithi192.stdout:Total 147 kB/s | 85 kB 00:00 2024-03-25T17:51:54.923 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-25T17:51:54.930 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-25T17:51:54.930 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-25T17:51:54.986 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-25T17:51:54.986 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-25T17:51:55.120 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-25T17:51:55.228 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:51:55.584 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-25T17:51:55.648 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:51:55.656 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T17:51:55.703 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T17:51:55.809 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T17:51:55.849 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T17:51:55.949 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:51:55.949 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:51:55.949 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-25T17:51:55.949 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:55.949 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:51:55.950 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T17:51:56.033 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T17:51:56.064 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T17:51:56.143 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T17:51:56.201 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install rbd-mirror 2024-03-25T17:51:56.773 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:05:41 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T17:51:56.844 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T17:51:56.927 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T17:51:56.928 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:51:56.928 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-03-25T17:51:56.928 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:51:56.928 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-25T17:51:56.928 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T17:51:56.928 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:51:56.928 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-25T17:51:56.928 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:51:56.929 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-03-25T17:51:56.929 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:51:56.929 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 3.1 M 2024-03-25T17:51:56.929 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 13 M 2024-03-25T17:51:56.929 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-25T17:51:56.962 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T17:51:58.275 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T17:51:59.356 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T17:51:59.388 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T17:52:02.097 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T17:52:02.098 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T17:52:02.098 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T17:52:02.098 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T17:52:02.098 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T17:52:02.098 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T17:52:02.098 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T17:52:02.098 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T17:52:02.098 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T17:52:02.098 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T17:52:02.098 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T17:52:02.098 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T17:52:02.570 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T17:52:02.570 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:02.570 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-25T17:52:02.570 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:52:02.570 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T17:52:02.570 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T17:52:02.570 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T17:52:02.570 INFO:teuthology.orchestra.run.smithi006.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T17:52:02.570 INFO:teuthology.orchestra.run.smithi006.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T17:52:02.570 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T17:52:02.570 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T17:52:02.571 INFO:teuthology.orchestra.run.smithi006.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T17:52:02.571 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T17:52:02.571 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T17:52:02.571 INFO:teuthology.orchestra.run.smithi006.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T17:52:02.571 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:02.571 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T17:52:02.934 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr-rook 2024-03-25T17:52:03.506 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:05:45 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T17:52:03.663 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T17:52:03.664 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================== 2024-03-25T17:52:03.664 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-03-25T17:52:03.664 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================== 2024-03-25T17:52:03.664 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-25T17:52:03.664 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T17:52:03.664 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-03-25T17:52:03.664 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T17:52:03.665 INFO:teuthology.orchestra.run.smithi006.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T17:52:03.665 INFO:teuthology.orchestra.run.smithi006.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T17:52:03.665 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T17:52:03.665 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T17:52:03.665 INFO:teuthology.orchestra.run.smithi006.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T17:52:03.665 INFO:teuthology.orchestra.run.smithi006.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T17:52:03.665 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T17:52:03.665 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T17:52:03.665 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T17:52:03.665 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T17:52:03.665 INFO:teuthology.orchestra.run.smithi006.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T17:52:03.665 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:03.665 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-25T17:52:03.665 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================== 2024-03-25T17:52:03.665 INFO:teuthology.orchestra.run.smithi006.stdout:Install 13 Packages 2024-03-25T17:52:03.666 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:03.666 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 2.0 M 2024-03-25T17:52:03.666 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 26 M 2024-03-25T17:52:03.666 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-25T17:52:04.006 INFO:teuthology.orchestra.run.smithi006.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 169 kB/s | 24 kB 00:00 2024-03-25T17:52:04.031 INFO:teuthology.orchestra.run.smithi006.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 107 kB/s | 18 kB 00:00 2024-03-25T17:52:04.090 INFO:teuthology.orchestra.run.smithi006.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.9 MB/s | 174 kB 00:00 2024-03-25T17:52:04.115 INFO:teuthology.orchestra.run.smithi006.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-25T17:52:04.141 INFO:teuthology.orchestra.run.smithi006.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.2 MB/s | 211 kB 00:00 2024-03-25T17:52:04.166 INFO:teuthology.orchestra.run.smithi006.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 859 kB/s | 43 kB 00:00 2024-03-25T17:52:04.224 INFO:teuthology.orchestra.run.smithi006.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 257 kB/s | 15 kB 00:00 2024-03-25T17:52:04.250 INFO:teuthology.orchestra.run.smithi006.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 296 kB/s | 32 kB 00:00 2024-03-25T17:52:04.300 INFO:teuthology.orchestra.run.smithi006.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 2.6 MB/s | 203 kB 00:00 2024-03-25T17:52:04.333 INFO:teuthology.orchestra.run.smithi006.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.7 MB/s | 59 kB 00:00 2024-03-25T17:52:04.367 INFO:teuthology.orchestra.run.smithi006.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 8.7 MB/s | 1.0 MB 00:00 2024-03-25T17:52:04.392 INFO:teuthology.orchestra.run.smithi006.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-03-25T17:52:04.451 INFO:teuthology.orchestra.run.smithi006.stdout:(13/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el 89 kB/s | 52 kB 00:00 2024-03-25T17:52:04.453 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:04.453 INFO:teuthology.orchestra.run.smithi006.stdout:Total 2.6 MB/s | 2.0 MB 00:00 2024-03-25T17:52:04.453 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-25T17:52:04.467 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-25T17:52:04.467 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-25T17:52:04.563 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-25T17:52:04.563 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-25T17:52:05.219 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-25T17:52:05.387 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T17:52:05.463 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T17:52:05.526 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T17:52:05.561 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T17:52:05.612 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T17:52:05.708 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T17:52:05.817 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T17:52:05.881 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T17:52:06.638 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T17:52:06.684 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T17:52:06.718 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T17:52:06.767 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T17:52:06.788 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T17:52:07.215 INFO:teuthology.orchestra.run.smithi192.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 307 kB/s | 3.1 MB 00:10 2024-03-25T17:52:07.216 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:07.216 INFO:teuthology.orchestra.run.smithi192.stdout:Total 307 kB/s | 3.1 MB 00:10 2024-03-25T17:52:07.217 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-25T17:52:07.228 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-25T17:52:07.228 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-25T17:52:07.305 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-25T17:52:07.305 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-25T17:52:07.644 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T17:52:07.645 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T17:52:07.645 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T17:52:07.645 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T17:52:07.645 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T17:52:07.645 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T17:52:07.645 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T17:52:07.645 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T17:52:07.645 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T17:52:07.645 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T17:52:07.645 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T17:52:07.645 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T17:52:07.645 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T17:52:07.741 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-25T17:52:07.799 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:07.824 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:07.824 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:52:07.824 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T17:52:07.824 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T17:52:07.824 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T17:52:07.824 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:52:08.015 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T17:52:08.016 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:08.016 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-25T17:52:08.016 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:52:08.016 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T17:52:08.016 INFO:teuthology.orchestra.run.smithi006.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T17:52:08.016 INFO:teuthology.orchestra.run.smithi006.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T17:52:08.016 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T17:52:08.016 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T17:52:08.016 INFO:teuthology.orchestra.run.smithi006.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T17:52:08.016 INFO:teuthology.orchestra.run.smithi006.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T17:52:08.016 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T17:52:08.016 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T17:52:08.017 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T17:52:08.017 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T17:52:08.017 INFO:teuthology.orchestra.run.smithi006.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T17:52:08.017 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:08.017 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T17:52:08.273 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T17:52:08.848 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:05:50 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T17:52:08.989 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:08.989 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:52:08.989 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-25T17:52:08.989 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:52:08.989 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:52:08.990 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T17:52:09.004 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T17:52:09.005 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================== 2024-03-25T17:52:09.005 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-03-25T17:52:09.005 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================== 2024-03-25T17:52:09.005 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-25T17:52:09.005 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T17:52:09.006 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-03-25T17:52:09.006 INFO:teuthology.orchestra.run.smithi006.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T17:52:09.006 INFO:teuthology.orchestra.run.smithi006.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T17:52:09.006 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T17:52:09.006 INFO:teuthology.orchestra.run.smithi006.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T17:52:09.006 INFO:teuthology.orchestra.run.smithi006.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T17:52:09.006 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:09.006 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-25T17:52:09.006 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================== 2024-03-25T17:52:09.006 INFO:teuthology.orchestra.run.smithi006.stdout:Install 6 Packages 2024-03-25T17:52:09.006 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:09.007 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 6.8 M 2024-03-25T17:52:09.007 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 33 M 2024-03-25T17:52:09.007 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-25T17:52:09.197 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install rbd-nbd 2024-03-25T17:52:09.360 INFO:teuthology.orchestra.run.smithi006.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.5 MB/s | 248 kB 00:00 2024-03-25T17:52:09.627 INFO:teuthology.orchestra.run.smithi006.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 13 MB/s | 5.8 MB 00:00 2024-03-25T17:52:09.773 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:05:54 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T17:52:09.861 INFO:teuthology.orchestra.run.smithi006.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 207 kB/s | 138 kB 00:00 2024-03-25T17:52:09.886 INFO:teuthology.orchestra.run.smithi006.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.0 MB/s | 548 kB 00:00 2024-03-25T17:52:09.903 INFO:teuthology.orchestra.run.smithi006.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 210 kB/s | 58 kB 00:00 2024-03-25T17:52:09.921 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T17:52:09.921 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:52:09.922 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2024-03-25T17:52:09.922 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:52:09.922 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-25T17:52:09.922 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T17:52:09.922 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:52:09.922 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-25T17:52:09.922 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:52:09.922 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-03-25T17:52:09.922 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:52:09.923 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 171 k 2024-03-25T17:52:09.923 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 494 k 2024-03-25T17:52:09.923 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-25T17:52:10.120 INFO:teuthology.orchestra.run.smithi006.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 197 kB/s | 51 kB 00:00 2024-03-25T17:52:10.123 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:10.123 INFO:teuthology.orchestra.run.smithi006.stdout:Total 6.1 MB/s | 6.8 MB 00:01 2024-03-25T17:52:10.136 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-25T17:52:10.146 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-25T17:52:10.146 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-25T17:52:10.239 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-25T17:52:10.240 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-25T17:52:10.554 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-25T17:52:10.726 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T17:52:10.789 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T17:52:11.227 INFO:teuthology.orchestra.run.smithi192.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 132 kB/s | 171 kB 00:01 2024-03-25T17:52:11.227 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:11.228 INFO:teuthology.orchestra.run.smithi192.stdout:Total 131 kB/s | 171 kB 00:01 2024-03-25T17:52:11.228 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-25T17:52:11.237 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-25T17:52:11.237 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-25T17:52:11.296 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-25T17:52:11.296 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-25T17:52:11.362 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T17:52:11.450 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-25T17:52:11.452 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T17:52:11.521 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T17:52:11.550 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:11.575 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T17:52:12.008 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:12.295 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:12.295 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:52:12.295 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-25T17:52:12.296 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:52:12.296 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:52:12.296 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T17:52:12.423 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T17:52:12.423 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T17:52:12.423 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T17:52:12.423 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T17:52:12.423 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T17:52:12.423 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T17:52:12.511 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:52:12.730 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T17:52:12.730 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:12.730 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-25T17:52:12.730 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:52:12.730 INFO:teuthology.orchestra.run.smithi006.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T17:52:12.730 INFO:teuthology.orchestra.run.smithi006.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T17:52:12.730 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T17:52:12.730 INFO:teuthology.orchestra.run.smithi006.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T17:52:12.730 INFO:teuthology.orchestra.run.smithi006.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T17:52:12.731 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:12.731 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T17:52:12.976 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-fuse 2024-03-25T17:52:13.084 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:05:58 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T17:52:13.239 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T17:52:13.239 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:52:13.239 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-03-25T17:52:13.240 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:52:13.240 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-25T17:52:13.240 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T17:52:13.240 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-03-25T17:52:13.240 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T17:52:13.240 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:52:13.240 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-25T17:52:13.240 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:52:13.240 INFO:teuthology.orchestra.run.smithi192.stdout:Install 2 Packages 2024-03-25T17:52:13.240 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:52:13.241 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 91 M 2024-03-25T17:52:13.241 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 498 M 2024-03-25T17:52:13.241 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-25T17:52:13.548 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:05:55 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T17:52:13.703 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T17:52:13.703 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:52:13.704 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-03-25T17:52:13.704 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:52:13.704 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-25T17:52:13.704 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T17:52:13.704 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-03-25T17:52:13.704 INFO:teuthology.orchestra.run.smithi006.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T17:52:13.704 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:13.704 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-25T17:52:13.704 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:52:13.704 INFO:teuthology.orchestra.run.smithi006.stdout:Install 2 Packages 2024-03-25T17:52:13.704 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:13.705 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 964 k 2024-03-25T17:52:13.705 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 2.8 M 2024-03-25T17:52:13.705 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-25T17:52:13.822 INFO:teuthology.orchestra.run.smithi006.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 674 kB/s | 79 kB 00:00 2024-03-25T17:52:15.831 INFO:teuthology.orchestra.run.smithi006.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 416 kB/s | 885 kB 00:02 2024-03-25T17:52:15.832 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:15.832 INFO:teuthology.orchestra.run.smithi006.stdout:Total 453 kB/s | 964 kB 00:02 2024-03-25T17:52:15.833 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-25T17:52:15.857 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-25T17:52:15.857 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-25T17:52:15.937 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-25T17:52:15.937 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-25T17:52:16.126 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-25T17:52:16.302 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T17:52:16.384 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T17:52:17.090 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T17:52:17.090 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T17:52:17.432 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T17:52:17.432 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:17.432 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-25T17:52:17.432 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T17:52:17.432 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:17.432 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T17:52:17.738 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-volume 2024-03-25T17:52:18.314 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:06:00 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T17:52:18.472 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T17:52:18.473 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:52:18.473 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-03-25T17:52:18.473 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:52:18.474 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-25T17:52:18.474 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T17:52:18.474 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-03-25T17:52:18.474 INFO:teuthology.orchestra.run.smithi006.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T17:52:18.474 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:18.474 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-25T17:52:18.474 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:52:18.474 INFO:teuthology.orchestra.run.smithi006.stdout:Install 2 Packages 2024-03-25T17:52:18.474 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:18.474 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 332 k 2024-03-25T17:52:18.474 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 1.5 M 2024-03-25T17:52:18.474 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-25T17:52:18.595 INFO:teuthology.orchestra.run.smithi006.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 571 kB/s | 69 kB 00:00 2024-03-25T17:52:20.120 INFO:teuthology.orchestra.run.smithi006.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 160 kB/s | 264 kB 00:01 2024-03-25T17:52:20.122 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:20.122 INFO:teuthology.orchestra.run.smithi006.stdout:Total 202 kB/s | 332 kB 00:01 2024-03-25T17:52:20.122 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-25T17:52:20.133 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-25T17:52:20.133 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-25T17:52:20.175 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-25T17:52:20.175 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-25T17:52:20.329 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-25T17:52:20.509 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T17:52:20.547 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T17:52:20.563 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T17:52:20.563 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:52:20.563 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T17:52:20.563 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:21.322 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T17:52:21.681 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T17:52:21.681 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:21.681 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-25T17:52:21.681 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:52:21.681 INFO:teuthology.orchestra.run.smithi006.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T17:52:21.681 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:21.681 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T17:52:22.032 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install librados-devel 2024-03-25T17:52:22.610 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:06:04 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T17:52:22.766 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T17:52:22.766 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:52:22.766 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-03-25T17:52:22.766 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:52:22.766 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-25T17:52:22.766 INFO:teuthology.orchestra.run.smithi006.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T17:52:22.766 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:22.766 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-25T17:52:22.767 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:52:22.767 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-03-25T17:52:22.767 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:22.767 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 127 k 2024-03-25T17:52:22.767 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 456 k 2024-03-25T17:52:22.767 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-25T17:52:23.526 INFO:teuthology.orchestra.run.smithi006.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 167 kB/s | 127 kB 00:00 2024-03-25T17:52:23.527 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:23.527 INFO:teuthology.orchestra.run.smithi006.stdout:Total 167 kB/s | 127 kB 00:00 2024-03-25T17:52:23.527 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-25T17:52:23.533 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-25T17:52:23.533 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-25T17:52:23.595 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-25T17:52:23.596 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-25T17:52:23.751 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-25T17:52:23.869 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:24.328 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:24.578 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:24.578 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:24.578 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-25T17:52:24.578 INFO:teuthology.orchestra.run.smithi006.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:52:24.578 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:24.579 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T17:52:24.855 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install libcephfs2 2024-03-25T17:52:25.429 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:06:07 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T17:52:25.529 INFO:teuthology.orchestra.run.smithi006.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:52:25.586 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T17:52:25.588 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-03-25T17:52:25.588 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T17:52:25.652 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install libcephfs-devel 2024-03-25T17:52:26.226 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:06:08 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T17:52:26.381 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T17:52:26.381 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:52:26.381 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-03-25T17:52:26.382 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:52:26.382 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-25T17:52:26.382 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T17:52:26.382 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:26.382 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-25T17:52:26.382 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:52:26.382 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-03-25T17:52:26.382 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:26.382 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 31 k 2024-03-25T17:52:26.382 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 140 k 2024-03-25T17:52:26.382 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-25T17:52:26.829 INFO:teuthology.orchestra.run.smithi006.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 70 kB/s | 31 kB 00:00 2024-03-25T17:52:26.830 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:26.830 INFO:teuthology.orchestra.run.smithi006.stdout:Total 70 kB/s | 31 kB 00:00 2024-03-25T17:52:26.830 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-25T17:52:26.833 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-25T17:52:26.834 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-25T17:52:26.866 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-25T17:52:26.866 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-25T17:52:26.987 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-25T17:52:27.097 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:27.477 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:27.753 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:27.754 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:27.754 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-25T17:52:27.754 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:52:27.754 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:27.754 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T17:52:27.954 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install librados2 2024-03-25T17:52:28.529 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:06:10 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T17:52:28.628 INFO:teuthology.orchestra.run.smithi006.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:52:28.683 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T17:52:28.684 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-03-25T17:52:28.684 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T17:52:28.739 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install librbd1 2024-03-25T17:52:29.315 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:06:11 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T17:52:29.417 INFO:teuthology.orchestra.run.smithi006.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:52:29.472 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T17:52:29.473 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-03-25T17:52:29.473 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T17:52:29.531 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install python3-rados 2024-03-25T17:52:30.110 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:06:12 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T17:52:30.210 INFO:teuthology.orchestra.run.smithi006.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:52:30.266 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T17:52:30.267 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-03-25T17:52:30.267 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T17:52:30.319 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install python3-rgw 2024-03-25T17:52:30.891 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:06:12 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T17:52:30.991 INFO:teuthology.orchestra.run.smithi006.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:52:31.045 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T17:52:31.046 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-03-25T17:52:31.046 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T17:52:31.101 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install python3-cephfs 2024-03-25T17:52:31.675 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:06:13 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T17:52:31.773 INFO:teuthology.orchestra.run.smithi006.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:52:31.828 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T17:52:31.829 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-03-25T17:52:31.829 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T17:52:31.887 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install python3-rbd 2024-03-25T17:52:32.457 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:06:14 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T17:52:32.561 INFO:teuthology.orchestra.run.smithi006.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:52:32.617 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T17:52:32.619 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-03-25T17:52:32.619 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T17:52:32.671 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install rbd-fuse 2024-03-25T17:52:33.242 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:06:15 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T17:52:33.395 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T17:52:33.396 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:52:33.396 INFO:teuthology.orchestra.run.smithi006.stdout: Package Architecture Version Repository Size 2024-03-25T17:52:33.396 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:52:33.396 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-25T17:52:33.396 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T17:52:33.396 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:33.396 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-25T17:52:33.397 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:52:33.397 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-03-25T17:52:33.397 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:33.397 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 85 k 2024-03-25T17:52:33.397 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 231 k 2024-03-25T17:52:33.397 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-25T17:52:34.066 INFO:teuthology.orchestra.run.smithi006.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 128 kB/s | 85 kB 00:00 2024-03-25T17:52:34.066 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:34.066 INFO:teuthology.orchestra.run.smithi006.stdout:Total 127 kB/s | 85 kB 00:00 2024-03-25T17:52:34.067 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-25T17:52:34.074 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-25T17:52:34.075 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-25T17:52:34.134 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-25T17:52:34.134 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-25T17:52:34.291 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-25T17:52:34.392 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:34.816 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:35.119 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:35.119 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:35.119 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-25T17:52:35.119 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:52:35.119 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:35.119 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T17:52:35.374 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install rbd-mirror 2024-03-25T17:52:35.949 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:06:17 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T17:52:36.105 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T17:52:36.105 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:52:36.105 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-03-25T17:52:36.105 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:52:36.105 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-25T17:52:36.105 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T17:52:36.105 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:36.106 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-25T17:52:36.106 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:52:36.106 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-03-25T17:52:36.106 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:36.106 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 3.1 M 2024-03-25T17:52:36.106 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 13 M 2024-03-25T17:52:36.106 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-25T17:52:46.226 INFO:teuthology.orchestra.run.smithi006.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 312 kB/s | 3.1 MB 00:10 2024-03-25T17:52:46.227 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:46.227 INFO:teuthology.orchestra.run.smithi006.stdout:Total 312 kB/s | 3.1 MB 00:10 2024-03-25T17:52:46.228 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-25T17:52:46.240 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-25T17:52:46.240 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-25T17:52:46.320 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-25T17:52:46.321 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-25T17:52:46.780 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-25T17:52:46.835 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:46.860 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:46.860 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:52:46.860 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T17:52:46.860 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T17:52:46.860 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T17:52:46.861 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:48.025 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:48.025 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:48.025 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-25T17:52:48.025 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:52:48.025 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:48.025 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T17:52:48.268 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install rbd-nbd 2024-03-25T17:52:48.845 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:06:30 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T17:52:48.998 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T17:52:48.999 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:52:48.999 INFO:teuthology.orchestra.run.smithi006.stdout: Package Architecture Version Repository Size 2024-03-25T17:52:48.999 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:52:48.999 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-25T17:52:49.000 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T17:52:49.000 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:49.000 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-25T17:52:49.000 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:52:49.000 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-03-25T17:52:49.000 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:49.000 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 171 k 2024-03-25T17:52:49.000 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 494 k 2024-03-25T17:52:49.000 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-25T17:52:50.121 INFO:teuthology.orchestra.run.smithi006.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 153 kB/s | 171 kB 00:01 2024-03-25T17:52:50.122 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:50.122 INFO:teuthology.orchestra.run.smithi006.stdout:Total 153 kB/s | 171 kB 00:01 2024-03-25T17:52:50.122 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-25T17:52:50.132 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-25T17:52:50.133 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-25T17:52:50.195 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-25T17:52:50.195 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-25T17:52:50.356 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-25T17:52:50.473 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:50.904 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:51.208 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:51.208 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:51.208 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-25T17:52:51.208 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:52:51.208 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:51.208 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T17:52:51.473 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:52:52.046 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:06:34 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T17:52:52.202 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T17:52:52.202 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:52:52.203 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-03-25T17:52:52.203 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:52:52.203 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-25T17:52:52.203 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T17:52:52.203 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-03-25T17:52:52.203 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T17:52:52.203 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:52.203 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-25T17:52:52.203 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:52:52.203 INFO:teuthology.orchestra.run.smithi006.stdout:Install 2 Packages 2024-03-25T17:52:52.203 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:52:52.203 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 91 M 2024-03-25T17:52:52.203 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 498 M 2024-03-25T17:52:52.203 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-25T17:53:13.456 INFO:teuthology.orchestra.run.smithi192.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 382 kB/s | 22 MB 01:00 2024-03-25T17:54:00.668 INFO:teuthology.orchestra.run.smithi006.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 336 kB/s | 22 MB 01:08 2024-03-25T17:54:23.998 INFO:teuthology.orchestra.run.smithi192.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 534 kB/s | 68 MB 02:10 2024-03-25T17:54:23.999 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:54:23.999 INFO:teuthology.orchestra.run.smithi192.stdout:Total 709 kB/s | 91 MB 02:10 2024-03-25T17:54:23.999 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-25T17:54:24.003 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-25T17:54:24.003 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-25T17:54:24.588 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-25T17:54:24.588 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-25T17:54:27.701 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-25T17:54:34.909 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T17:54:35.064 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T17:54:38.467 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T17:54:38.468 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T17:54:38.748 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T17:54:38.749 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:54:38.749 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-25T17:54:38.749 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:54:38.749 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:54:38.749 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:54:38.749 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T17:54:38.973 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-common-debuginfo 2024-03-25T17:54:39.547 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:08:24 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T17:54:39.695 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T17:54:39.696 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:54:39.696 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-03-25T17:54:39.696 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:54:39.696 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-25T17:54:39.696 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T17:54:39.696 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:54:39.696 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-25T17:54:39.696 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T17:54:39.696 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-03-25T17:54:39.697 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T17:54:39.697 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 416 M 2024-03-25T17:54:39.697 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 1.7 G 2024-03-25T17:54:39.697 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-25T17:56:21.481 INFO:teuthology.orchestra.run.smithi006.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 333 kB/s | 68 MB 03:29 2024-03-25T17:56:21.482 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:56:21.482 INFO:teuthology.orchestra.run.smithi006.stdout:Total 443 kB/s | 91 MB 03:29 2024-03-25T17:56:21.482 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-25T17:56:21.486 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-25T17:56:21.487 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-25T17:56:22.101 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-25T17:56:22.102 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-25T17:56:25.385 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-25T17:56:32.925 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T17:56:33.000 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T17:56:36.348 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T17:56:36.348 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T17:56:36.623 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T17:56:36.623 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:56:36.623 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-25T17:56:36.623 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:56:36.623 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:56:36.624 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:56:36.624 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T17:56:36.850 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-common-debuginfo 2024-03-25T17:56:37.424 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:10:19 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T17:56:37.578 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T17:56:37.579 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:56:37.579 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-03-25T17:56:37.579 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:56:37.579 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-25T17:56:37.579 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T17:56:37.579 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:56:37.579 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-25T17:56:37.579 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T17:56:37.579 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-03-25T17:56:37.579 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T17:56:37.579 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 416 M 2024-03-25T17:56:37.580 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 1.7 G 2024-03-25T17:56:37.580 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-25T18:11:02.736 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 433 kB/s | 416 MB 16:23 2024-03-25T18:11:02.736 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:11:02.736 INFO:teuthology.orchestra.run.smithi192.stdout:Total 433 kB/s | 416 MB 16:23 2024-03-25T18:11:02.736 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-25T18:11:02.739 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-25T18:11:02.739 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-25T18:11:05.202 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-25T18:11:05.202 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-25T18:11:05.806 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 491 kB/s | 416 MB 14:28 2024-03-25T18:11:05.807 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:11:05.807 INFO:teuthology.orchestra.run.smithi006.stdout:Total 491 kB/s | 416 MB 14:28 2024-03-25T18:11:05.807 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-25T18:11:05.812 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-25T18:11:05.812 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-25T18:11:08.399 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-25T18:11:08.399 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-25T18:11:47.227 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-25T18:11:47.329 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:11:53.235 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-25T18:11:53.338 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:11:54.353 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:11:54.619 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:11:54.619 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T18:11:54.619 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-25T18:11:54.619 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:11:54.619 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T18:11:54.619 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T18:11:54.854 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T18:11:55.416 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:25:40 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T18:11:55.566 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T18:11:55.567 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================== 2024-03-25T18:11:55.567 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-03-25T18:11:55.567 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================== 2024-03-25T18:11:55.567 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-25T18:11:55.567 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T18:11:55.568 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T18:11:55.568 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-25T18:11:55.568 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================== 2024-03-25T18:11:55.568 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-03-25T18:11:55.568 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T18:11:55.568 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 1.4 M 2024-03-25T18:11:55.568 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 6.4 M 2024-03-25T18:11:55.568 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-25T18:11:59.874 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 339 kB/s | 1.4 MB 00:04 2024-03-25T18:11:59.875 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:11:59.876 INFO:teuthology.orchestra.run.smithi192.stdout:Total 339 kB/s | 1.4 MB 00:04 2024-03-25T18:11:59.876 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-25T18:11:59.880 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-25T18:11:59.880 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-25T18:11:59.923 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-25T18:11:59.924 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-25T18:12:00.222 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-25T18:12:00.340 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:12:00.741 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:12:01.030 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:12:01.030 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T18:12:01.030 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-25T18:12:01.031 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:12:01.031 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T18:12:01.031 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T18:12:01.204 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T18:12:01.287 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:12:01.581 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:12:01.581 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T18:12:01.581 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-25T18:12:01.581 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:12:01.581 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T18:12:01.581 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T18:12:01.763 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:25:46 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T18:12:01.845 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T18:12:01.912 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T18:12:01.912 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T18:12:01.912 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-03-25T18:12:01.912 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T18:12:01.912 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-25T18:12:01.912 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T18:12:01.913 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T18:12:01.913 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-25T18:12:01.913 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T18:12:01.913 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-03-25T18:12:01.913 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T18:12:01.913 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 176 M 2024-03-25T18:12:01.913 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 753 M 2024-03-25T18:12:01.914 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-25T18:12:02.421 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:25:44 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T18:12:02.576 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T18:12:02.576 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================== 2024-03-25T18:12:02.576 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-03-25T18:12:02.576 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================== 2024-03-25T18:12:02.577 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-25T18:12:02.577 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T18:12:02.577 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T18:12:02.577 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-25T18:12:02.577 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================== 2024-03-25T18:12:02.577 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-03-25T18:12:02.577 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T18:12:02.578 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 1.4 M 2024-03-25T18:12:02.578 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 6.4 M 2024-03-25T18:12:02.578 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-25T18:12:07.439 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 301 kB/s | 1.4 MB 00:04 2024-03-25T18:12:07.440 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:12:07.440 INFO:teuthology.orchestra.run.smithi006.stdout:Total 300 kB/s | 1.4 MB 00:04 2024-03-25T18:12:07.440 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-25T18:12:07.444 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-25T18:12:07.444 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-25T18:12:07.492 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-25T18:12:07.493 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-25T18:12:07.763 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-25T18:12:07.874 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:12:08.286 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:12:08.564 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:12:08.564 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T18:12:08.564 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-25T18:12:08.564 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:12:08.564 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T18:12:08.564 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T18:12:08.766 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T18:12:09.337 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:25:51 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T18:12:09.491 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T18:12:09.491 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T18:12:09.491 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-03-25T18:12:09.491 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T18:12:09.491 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-25T18:12:09.492 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T18:12:09.492 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T18:12:09.492 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-25T18:12:09.492 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T18:12:09.492 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-03-25T18:12:09.492 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T18:12:09.492 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 176 M 2024-03-25T18:12:09.492 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 753 M 2024-03-25T18:12:09.492 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-25T18:17:57.750 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 506 kB/s | 176 MB 05:55 2024-03-25T18:17:57.751 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:17:57.751 INFO:teuthology.orchestra.run.smithi192.stdout:Total 506 kB/s | 176 MB 05:55 2024-03-25T18:17:57.751 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-25T18:17:57.754 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-25T18:17:57.754 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-25T18:17:58.810 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-25T18:17:58.810 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-25T18:18:17.183 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-25T18:18:17.345 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:18:22.007 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:18:22.268 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:18:22.268 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T18:18:22.268 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-25T18:18:22.268 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:18:22.268 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T18:18:22.268 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T18:18:22.487 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-test-debuginfo 2024-03-25T18:18:23.053 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:32:08 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T18:18:23.201 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T18:18:23.202 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T18:18:23.202 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-03-25T18:18:23.202 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T18:18:23.202 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-25T18:18:23.202 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T18:18:23.202 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T18:18:23.202 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-25T18:18:23.202 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T18:18:23.202 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-03-25T18:18:23.203 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T18:18:23.203 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 672 M 2024-03-25T18:18:23.203 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 3.1 G 2024-03-25T18:18:23.203 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-25T18:19:42.090 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 397 kB/s | 176 MB 07:32 2024-03-25T18:19:42.091 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:19:42.091 INFO:teuthology.orchestra.run.smithi006.stdout:Total 397 kB/s | 176 MB 07:32 2024-03-25T18:19:42.091 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-25T18:19:42.094 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-25T18:19:42.094 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-25T18:19:43.204 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-25T18:19:43.205 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-25T18:20:02.318 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-25T18:20:02.422 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:20:07.115 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:20:07.375 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:20:07.375 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T18:20:07.375 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-25T18:20:07.375 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:07.375 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T18:20:07.376 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T18:20:07.583 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-test-debuginfo 2024-03-25T18:20:08.156 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:33:50 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T18:20:08.313 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T18:20:08.313 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T18:20:08.314 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-03-25T18:20:08.314 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T18:20:08.314 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-25T18:20:08.314 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T18:20:08.314 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T18:20:08.314 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-25T18:20:08.314 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T18:20:08.314 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-03-25T18:20:08.314 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T18:20:08.314 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 672 M 2024-03-25T18:20:08.314 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 3.1 G 2024-03-25T18:20:08.314 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-25T18:45:25.788 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 424 kB/s | 672 MB 27:02 2024-03-25T18:45:25.789 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:45:25.789 INFO:teuthology.orchestra.run.smithi192.stdout:Total 424 kB/s | 672 MB 27:02 2024-03-25T18:45:25.789 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-25T18:45:25.793 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-25T18:45:25.793 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-25T18:45:29.764 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-25T18:45:29.764 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-25T18:46:40.333 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-25T18:46:40.858 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:46:48.815 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:46:49.078 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:46:49.078 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T18:46:49.078 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-25T18:46:49.078 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:46:49.078 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T18:46:49.078 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T18:46:49.371 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:46:49.948 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 1:00:34 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T18:46:50.050 INFO:teuthology.orchestra.run.smithi192.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:46:50.106 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T18:46:50.107 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-03-25T18:46:50.107 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T18:46:50.187 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T18:46:50.764 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 1:00:35 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T18:46:50.914 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T18:46:50.915 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T18:46:50.915 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-03-25T18:46:50.915 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T18:46:50.915 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-25T18:46:50.915 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:46:50.916 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T18:46:50.916 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-25T18:46:50.916 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T18:46:50.916 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-03-25T18:46:50.916 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T18:46:50.916 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 20 M 2024-03-25T18:46:50.916 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 103 M 2024-03-25T18:46:50.916 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-25T18:47:40.179 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 409 kB/s | 20 MB 00:49 2024-03-25T18:47:40.180 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:47:40.180 INFO:teuthology.orchestra.run.smithi192.stdout:Total 409 kB/s | 20 MB 00:49 2024-03-25T18:47:40.180 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-25T18:47:40.184 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-25T18:47:40.184 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-25T18:47:40.333 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-25T18:47:40.333 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-25T18:47:42.575 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-25T18:47:42.826 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:47:43.726 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:47:43.985 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:47:43.985 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T18:47:43.985 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-25T18:47:43.985 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:47:43.985 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T18:47:43.986 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T18:47:44.149 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T18:47:44.715 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 1:01:29 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T18:47:44.863 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T18:47:44.864 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T18:47:44.864 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-03-25T18:47:44.864 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T18:47:44.864 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-25T18:47:44.864 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T18:47:44.864 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T18:47:44.865 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-25T18:47:44.865 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T18:47:44.865 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-03-25T18:47:44.865 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T18:47:44.865 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 48 M 2024-03-25T18:47:44.865 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 173 M 2024-03-25T18:47:44.865 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-25T18:47:55.387 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 413 kB/s | 672 MB 27:47 2024-03-25T18:47:55.387 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:47:55.388 INFO:teuthology.orchestra.run.smithi006.stdout:Total 413 kB/s | 672 MB 27:47 2024-03-25T18:47:55.388 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-25T18:47:55.392 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-25T18:47:55.393 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-25T18:47:59.544 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-25T18:47:59.544 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-25T18:49:14.623 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-25T18:49:15.160 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:49:20.069 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 513 kB/s | 48 MB 01:35 2024-03-25T18:49:20.071 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:49:20.071 INFO:teuthology.orchestra.run.smithi192.stdout:Total 513 kB/s | 48 MB 01:35 2024-03-25T18:49:20.071 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-25T18:49:20.074 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-25T18:49:20.074 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-25T18:49:20.389 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-25T18:49:20.390 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-25T18:49:22.738 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:49:23.023 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:49:23.023 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T18:49:23.023 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-25T18:49:23.023 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:49:23.023 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T18:49:23.023 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T18:49:23.306 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:49:23.880 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 1:03:05 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T18:49:23.979 INFO:teuthology.orchestra.run.smithi006.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:49:24.034 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T18:49:24.035 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-03-25T18:49:24.036 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T18:49:24.092 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T18:49:24.665 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 1:03:06 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T18:49:24.819 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T18:49:24.820 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T18:49:24.820 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-03-25T18:49:24.820 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T18:49:24.820 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-25T18:49:24.820 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:49:24.820 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T18:49:24.820 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-25T18:49:24.820 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T18:49:24.820 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-03-25T18:49:24.821 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T18:49:24.821 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 20 M 2024-03-25T18:49:24.821 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 103 M 2024-03-25T18:49:24.821 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-25T18:49:25.159 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-25T18:49:25.251 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:49:26.580 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:49:26.846 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:49:26.846 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T18:49:26.846 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-25T18:49:26.846 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:49:26.846 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T18:49:26.846 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T18:49:27.018 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T18:49:27.585 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 1:03:12 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T18:49:27.734 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T18:49:27.735 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T18:49:27.735 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-03-25T18:49:27.735 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T18:49:27.735 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-25T18:49:27.735 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T18:49:27.735 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T18:49:27.735 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-25T18:49:27.735 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T18:49:27.736 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-03-25T18:49:27.736 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T18:49:27.736 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 99 M 2024-03-25T18:49:27.736 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 449 M 2024-03-25T18:49:27.736 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-25T18:50:16.930 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 387 kB/s | 20 MB 00:52 2024-03-25T18:50:16.931 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:50:16.931 INFO:teuthology.orchestra.run.smithi006.stdout:Total 387 kB/s | 20 MB 00:52 2024-03-25T18:50:16.931 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-25T18:50:16.935 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-25T18:50:16.935 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-25T18:50:17.091 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-25T18:50:17.092 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-25T18:50:19.447 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-25T18:50:19.564 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:50:20.477 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:50:20.858 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:50:20.859 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T18:50:20.859 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-25T18:50:20.859 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:50:20.859 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T18:50:20.859 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T18:50:21.022 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T18:50:21.595 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 1:04:03 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T18:50:21.749 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T18:50:21.750 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T18:50:21.750 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-03-25T18:50:21.750 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T18:50:21.750 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-25T18:50:21.750 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T18:50:21.750 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T18:50:21.750 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-25T18:50:21.751 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T18:50:21.751 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-03-25T18:50:21.751 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T18:50:21.751 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 48 M 2024-03-25T18:50:21.752 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 173 M 2024-03-25T18:50:21.752 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-25T18:51:44.258 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 592 kB/s | 48 MB 01:22 2024-03-25T18:51:44.259 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:51:44.259 INFO:teuthology.orchestra.run.smithi006.stdout:Total 592 kB/s | 48 MB 01:22 2024-03-25T18:51:44.259 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-25T18:51:44.263 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-25T18:51:44.263 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-25T18:51:44.589 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-25T18:51:44.590 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-25T18:51:49.610 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-25T18:51:49.700 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:51:51.144 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:51:51.409 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:51:51.409 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T18:51:51.409 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-25T18:51:51.409 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:51:51.410 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T18:51:51.410 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T18:51:51.570 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T18:51:52.142 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 1:05:34 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T18:51:52.296 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T18:51:52.296 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T18:51:52.297 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-03-25T18:51:52.297 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T18:51:52.297 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-25T18:51:52.297 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T18:51:52.297 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T18:51:52.297 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-25T18:51:52.297 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T18:51:52.297 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-03-25T18:51:52.297 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T18:51:52.298 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 99 M 2024-03-25T18:51:52.298 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 449 M 2024-03-25T18:51:52.298 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-25T18:54:18.947 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 688 kB/s | 99 MB 02:26 2024-03-25T18:54:18.947 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:54:18.948 INFO:teuthology.orchestra.run.smithi006.stdout:Total 688 kB/s | 99 MB 02:26 2024-03-25T18:54:18.948 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-25T18:54:18.951 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-25T18:54:18.951 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-25T18:54:19.588 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-25T18:54:19.588 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-25T18:54:30.495 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-25T18:54:30.587 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:54:33.605 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:54:33.872 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:54:33.873 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T18:54:33.873 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-25T18:54:33.873 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:54:33.873 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T18:54:33.873 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T18:54:34.159 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T18:54:34.732 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 1:08:16 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T18:54:34.886 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T18:54:34.887 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T18:54:34.887 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-03-25T18:54:34.887 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T18:54:34.887 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-25T18:54:34.888 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T18:54:34.888 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T18:54:34.888 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-25T18:54:34.888 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T18:54:34.888 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-03-25T18:54:34.888 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T18:54:34.888 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 287 M 2024-03-25T18:54:34.888 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 1.2 G 2024-03-25T18:54:34.888 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-25T18:54:57.090 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 306 kB/s | 99 MB 05:29 2024-03-25T18:54:57.091 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:54:57.091 INFO:teuthology.orchestra.run.smithi192.stdout:Total 306 kB/s | 99 MB 05:29 2024-03-25T18:54:57.091 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-25T18:54:57.094 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-25T18:54:57.094 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-25T18:54:57.702 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-25T18:54:57.702 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-25T18:55:07.966 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-25T18:55:08.061 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:55:10.962 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:55:11.227 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:55:11.228 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T18:55:11.228 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-25T18:55:11.228 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:55:11.228 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T18:55:11.228 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T18:55:11.430 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T18:55:11.991 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 1:08:56 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T18:55:12.140 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T18:55:12.141 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T18:55:12.141 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-03-25T18:55:12.142 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T18:55:12.142 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-25T18:55:12.142 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T18:55:12.142 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T18:55:12.142 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-25T18:55:12.142 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T18:55:12.142 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-03-25T18:55:12.142 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T18:55:12.142 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 287 M 2024-03-25T18:55:12.142 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 1.2 G 2024-03-25T18:55:12.142 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-25T19:02:56.607 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 585 kB/s | 287 MB 08:21 2024-03-25T19:02:56.608 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:02:56.608 INFO:teuthology.orchestra.run.smithi006.stdout:Total 585 kB/s | 287 MB 08:21 2024-03-25T19:02:56.608 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-25T19:02:56.611 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-25T19:02:56.611 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-25T19:02:58.402 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-25T19:02:58.402 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-25T19:03:29.480 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-25T19:03:29.882 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:03:36.312 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:03:36.584 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:03:36.585 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:03:36.585 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-25T19:03:36.585 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:03:36.585 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:03:36.585 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T19:03:36.782 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T19:03:37.357 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 1:17:19 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T19:03:37.513 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T19:03:37.514 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T19:03:37.514 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-03-25T19:03:37.514 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T19:03:37.514 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-25T19:03:37.514 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T19:03:37.514 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:03:37.514 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-25T19:03:37.514 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T19:03:37.514 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-03-25T19:03:37.514 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:03:37.514 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 5.9 M 2024-03-25T19:03:37.514 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 28 M 2024-03-25T19:03:37.515 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-25T19:03:50.972 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 452 kB/s | 5.9 MB 00:13 2024-03-25T19:03:50.973 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:03:50.973 INFO:teuthology.orchestra.run.smithi006.stdout:Total 452 kB/s | 5.9 MB 00:13 2024-03-25T19:03:50.974 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-25T19:03:50.978 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-25T19:03:50.978 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-25T19:03:51.053 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-25T19:03:51.053 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-25T19:03:51.831 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-25T19:03:51.940 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:03:52.644 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:03:52.922 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:03:52.923 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:03:52.923 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-25T19:03:52.923 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:03:52.923 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:03:52.923 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T19:03:53.139 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install librados-devel-debuginfo 2024-03-25T19:03:53.716 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 1:17:35 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T19:03:53.874 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T19:03:53.875 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T19:03:53.875 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-03-25T19:03:53.875 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T19:03:53.875 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-25T19:03:53.875 INFO:teuthology.orchestra.run.smithi006.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T19:03:53.875 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:03:53.875 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-25T19:03:53.875 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T19:03:53.875 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-03-25T19:03:53.875 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:03:53.875 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 109 k 2024-03-25T19:03:53.875 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 440 k 2024-03-25T19:03:53.875 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-25T19:03:54.609 INFO:teuthology.orchestra.run.smithi006.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 148 kB/s | 109 kB 00:00 2024-03-25T19:03:54.609 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:03:54.610 INFO:teuthology.orchestra.run.smithi006.stdout:Total 148 kB/s | 109 kB 00:00 2024-03-25T19:03:54.610 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-25T19:03:54.613 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-25T19:03:54.613 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-25T19:03:54.651 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-25T19:03:54.651 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-25T19:03:54.772 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-25T19:03:54.885 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:03:55.258 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:03:55.529 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:03:55.530 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:03:55.530 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-25T19:03:55.530 INFO:teuthology.orchestra.run.smithi006.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:03:55.530 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:03:55.530 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T19:03:55.719 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T19:03:56.296 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 1:17:38 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T19:03:56.451 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T19:03:56.451 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T19:03:56.451 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-03-25T19:03:56.451 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T19:03:56.451 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-25T19:03:56.451 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T19:03:56.451 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:03:56.451 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-25T19:03:56.451 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T19:03:56.452 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-03-25T19:03:56.452 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:03:56.452 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 5.4 M 2024-03-25T19:03:56.452 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 25 M 2024-03-25T19:03:56.452 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-25T19:04:14.610 INFO:teuthology.orchestra.run.smithi006.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 302 kB/s | 5.4 MB 00:18 2024-03-25T19:04:14.610 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:04:14.611 INFO:teuthology.orchestra.run.smithi006.stdout:Total 302 kB/s | 5.4 MB 00:18 2024-03-25T19:04:14.611 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-25T19:04:14.615 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-25T19:04:14.615 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-25T19:04:14.687 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-25T19:04:14.687 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-25T19:04:15.401 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-25T19:04:15.528 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:04:16.199 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:04:16.470 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:04:16.470 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:04:16.470 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-25T19:04:16.470 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:04:16.470 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:04:16.470 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T19:04:16.777 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install librados2-debuginfo 2024-03-25T19:04:17.350 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 1:17:59 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T19:04:17.504 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T19:04:17.505 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T19:04:17.505 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-03-25T19:04:17.505 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T19:04:17.505 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-25T19:04:17.505 INFO:teuthology.orchestra.run.smithi006.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T19:04:17.505 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:04:17.505 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-25T19:04:17.505 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T19:04:17.505 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-03-25T19:04:17.506 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:04:17.506 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 44 M 2024-03-25T19:04:17.506 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 253 M 2024-03-25T19:04:17.506 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-25T19:04:36.004 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 521 kB/s | 287 MB 09:23 2024-03-25T19:04:36.005 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:04:36.005 INFO:teuthology.orchestra.run.smithi192.stdout:Total 521 kB/s | 287 MB 09:23 2024-03-25T19:04:36.005 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-25T19:04:36.008 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-25T19:04:36.009 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-25T19:04:37.713 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-25T19:04:37.713 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-25T19:05:07.049 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-25T19:05:07.618 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:05:10.175 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:05:10.446 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:05:10.446 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:05:10.446 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-25T19:05:10.446 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:05:10.446 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:05:10.446 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T19:05:10.656 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T19:05:11.227 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 1:18:56 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T19:05:11.379 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T19:05:11.380 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T19:05:11.380 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-03-25T19:05:11.380 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T19:05:11.380 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-25T19:05:11.380 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T19:05:11.380 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:05:11.380 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-25T19:05:11.381 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T19:05:11.381 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-03-25T19:05:11.381 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:05:11.381 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 5.9 M 2024-03-25T19:05:11.381 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 28 M 2024-03-25T19:05:11.381 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-25T19:05:26.678 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 398 kB/s | 5.9 MB 00:15 2024-03-25T19:05:26.679 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:05:26.679 INFO:teuthology.orchestra.run.smithi192.stdout:Total 398 kB/s | 5.9 MB 00:15 2024-03-25T19:05:26.679 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-25T19:05:26.683 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-25T19:05:26.683 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-25T19:05:26.756 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-25T19:05:26.756 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-25T19:05:27.495 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-25T19:05:27.602 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:05:28.173 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:05:28.435 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:05:28.435 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:05:28.435 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-25T19:05:28.435 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:05:28.435 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:05:28.435 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T19:05:28.653 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install librados-devel-debuginfo 2024-03-25T19:05:29.217 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 1:19:14 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T19:05:29.368 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T19:05:29.368 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T19:05:29.369 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-03-25T19:05:29.369 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T19:05:29.369 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-25T19:05:29.369 INFO:teuthology.orchestra.run.smithi192.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T19:05:29.369 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:05:29.369 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-25T19:05:29.369 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T19:05:29.369 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-03-25T19:05:29.369 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:05:29.370 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 109 k 2024-03-25T19:05:29.370 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 440 k 2024-03-25T19:05:29.370 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-25T19:05:30.523 INFO:teuthology.orchestra.run.smithi192.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 94 kB/s | 109 kB 00:01 2024-03-25T19:05:30.523 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:05:30.524 INFO:teuthology.orchestra.run.smithi192.stdout:Total 94 kB/s | 109 kB 00:01 2024-03-25T19:05:30.524 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-25T19:05:30.528 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-25T19:05:30.528 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-25T19:05:30.566 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-25T19:05:30.566 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-25T19:05:30.688 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-25T19:05:30.805 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:05:31.190 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:05:31.460 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:05:31.460 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:05:31.460 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-25T19:05:31.461 INFO:teuthology.orchestra.run.smithi192.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:05:31.461 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:05:31.461 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T19:05:31.657 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T19:05:32.222 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 1:19:17 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T19:05:32.371 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T19:05:32.371 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T19:05:32.371 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-03-25T19:05:32.371 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T19:05:32.371 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-25T19:05:32.371 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T19:05:32.372 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:05:32.372 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-25T19:05:32.372 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T19:05:32.372 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-03-25T19:05:32.372 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:05:32.372 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 5.4 M 2024-03-25T19:05:32.372 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 25 M 2024-03-25T19:05:32.373 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-25T19:05:45.816 INFO:teuthology.orchestra.run.smithi192.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 408 kB/s | 5.4 MB 00:13 2024-03-25T19:05:45.817 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:05:45.817 INFO:teuthology.orchestra.run.smithi192.stdout:Total 408 kB/s | 5.4 MB 00:13 2024-03-25T19:05:45.817 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-25T19:05:45.821 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-25T19:05:45.821 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-25T19:05:45.890 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-25T19:05:45.890 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-25T19:05:46.557 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-25T19:05:46.673 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:05:47.213 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:05:47.474 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:05:47.474 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:05:47.474 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-25T19:05:47.474 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:05:47.474 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:05:47.474 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T19:05:47.683 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install librados2-debuginfo 2024-03-25T19:05:48.249 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 1:19:33 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T19:05:48.398 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T19:05:48.398 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T19:05:48.399 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-03-25T19:05:48.399 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T19:05:48.399 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-25T19:05:48.399 INFO:teuthology.orchestra.run.smithi192.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T19:05:48.399 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:05:48.399 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-25T19:05:48.399 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T19:05:48.399 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-03-25T19:05:48.399 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:05:48.400 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 44 M 2024-03-25T19:05:48.400 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 253 M 2024-03-25T19:05:48.400 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-25T19:06:06.460 INFO:teuthology.orchestra.run.smithi006.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 416 kB/s | 44 MB 01:48 2024-03-25T19:06:06.461 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:06:06.461 INFO:teuthology.orchestra.run.smithi006.stdout:Total 416 kB/s | 44 MB 01:48 2024-03-25T19:06:06.462 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-25T19:06:06.465 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-25T19:06:06.465 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-25T19:06:06.777 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-25T19:06:06.778 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-25T19:06:12.133 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-25T19:06:12.237 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:06:14.166 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:06:14.446 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:06:14.446 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:06:14.446 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-25T19:06:14.446 INFO:teuthology.orchestra.run.smithi006.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:06:14.446 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:06:14.446 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T19:06:14.632 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install librbd1-debuginfo 2024-03-25T19:06:15.205 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 1:19:57 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T19:06:15.359 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T19:06:15.360 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T19:06:15.360 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-03-25T19:06:15.360 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T19:06:15.360 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-25T19:06:15.360 INFO:teuthology.orchestra.run.smithi006.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:06:15.360 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:06:15.360 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-25T19:06:15.360 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T19:06:15.360 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-03-25T19:06:15.361 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:06:15.361 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 52 M 2024-03-25T19:06:15.361 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 308 M 2024-03-25T19:06:15.361 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-25T19:08:02.253 INFO:teuthology.orchestra.run.smithi192.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 339 kB/s | 44 MB 02:13 2024-03-25T19:08:02.254 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:08:02.254 INFO:teuthology.orchestra.run.smithi192.stdout:Total 339 kB/s | 44 MB 02:13 2024-03-25T19:08:02.255 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-25T19:08:02.257 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-25T19:08:02.257 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-25T19:08:02.561 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-25T19:08:02.561 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-25T19:08:07.688 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-25T19:08:07.801 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:08:09.677 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:08:09.950 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:08:09.950 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:08:09.950 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-25T19:08:09.950 INFO:teuthology.orchestra.run.smithi192.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:08:09.950 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:08:09.950 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T19:08:10.169 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install librbd1-debuginfo 2024-03-25T19:08:10.736 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 1:21:55 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T19:08:10.886 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T19:08:10.887 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T19:08:10.887 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-03-25T19:08:10.887 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T19:08:10.887 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-25T19:08:10.887 INFO:teuthology.orchestra.run.smithi192.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:08:10.887 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:08:10.887 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-25T19:08:10.888 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T19:08:10.888 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-03-25T19:08:10.888 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:08:10.888 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 52 M 2024-03-25T19:08:10.888 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 308 M 2024-03-25T19:08:10.888 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-25T19:08:48.648 INFO:teuthology.orchestra.run.smithi006.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 346 kB/s | 52 MB 02:33 2024-03-25T19:08:48.649 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:08:48.649 INFO:teuthology.orchestra.run.smithi006.stdout:Total 346 kB/s | 52 MB 02:33 2024-03-25T19:08:48.649 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-25T19:08:48.653 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-25T19:08:48.653 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-25T19:08:49.012 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-25T19:08:49.012 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-25T19:08:55.367 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-25T19:08:55.475 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:08:57.597 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:08:57.975 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:08:57.975 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:08:57.975 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-25T19:08:57.975 INFO:teuthology.orchestra.run.smithi006.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:08:57.975 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:08:57.975 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T19:08:58.152 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T19:08:58.721 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 1:22:40 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T19:08:58.877 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T19:08:58.878 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T19:08:58.878 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-03-25T19:08:58.878 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T19:08:58.878 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-25T19:08:58.878 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T19:08:58.878 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:08:58.878 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-25T19:08:58.879 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T19:08:58.879 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-03-25T19:08:58.879 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:08:58.879 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 615 k 2024-03-25T19:08:58.879 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 2.1 M 2024-03-25T19:08:58.879 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-25T19:09:00.201 INFO:teuthology.orchestra.run.smithi006.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 465 kB/s | 615 kB 00:01 2024-03-25T19:09:00.201 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:09:00.202 INFO:teuthology.orchestra.run.smithi006.stdout:Total 464 kB/s | 615 kB 00:01 2024-03-25T19:09:00.202 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-25T19:09:00.206 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-25T19:09:00.206 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-25T19:09:00.249 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-25T19:09:00.249 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-25T19:09:00.430 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-25T19:09:00.527 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:09:00.928 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:09:01.208 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:09:01.208 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:09:01.208 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-25T19:09:01.208 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:09:01.208 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:09:01.209 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T19:09:01.415 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install python3-rados-debuginfo 2024-03-25T19:09:01.992 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 1:22:43 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T19:09:02.152 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T19:09:02.153 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T19:09:02.153 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-03-25T19:09:02.153 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T19:09:02.153 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-25T19:09:02.153 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T19:09:02.153 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:09:02.153 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-25T19:09:02.153 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T19:09:02.153 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-03-25T19:09:02.153 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:09:02.153 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 771 k 2024-03-25T19:09:02.153 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 2.7 M 2024-03-25T19:09:02.153 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-25T19:09:03.637 INFO:teuthology.orchestra.run.smithi006.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 520 kB/s | 771 kB 00:01 2024-03-25T19:09:03.638 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:09:03.638 INFO:teuthology.orchestra.run.smithi006.stdout:Total 519 kB/s | 771 kB 00:01 2024-03-25T19:09:03.638 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-25T19:09:03.642 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-25T19:09:03.642 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-25T19:09:03.686 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-25T19:09:03.686 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-25T19:09:03.879 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-25T19:09:03.989 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:09:04.396 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:09:04.658 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:09:04.658 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:09:04.658 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-25T19:09:04.658 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:09:04.658 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:09:04.658 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T19:09:04.869 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T19:09:05.445 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 1:22:47 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T19:09:05.600 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T19:09:05.601 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T19:09:05.601 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-03-25T19:09:05.601 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T19:09:05.601 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-25T19:09:05.601 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T19:09:05.601 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:09:05.601 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-25T19:09:05.602 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T19:09:05.602 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-03-25T19:09:05.602 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:09:05.602 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 1.1 M 2024-03-25T19:09:05.602 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 4.3 M 2024-03-25T19:09:05.602 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-25T19:09:10.140 INFO:teuthology.orchestra.run.smithi006.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 254 kB/s | 1.1 MB 00:04 2024-03-25T19:09:10.141 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:09:10.141 INFO:teuthology.orchestra.run.smithi006.stdout:Total 254 kB/s | 1.1 MB 00:04 2024-03-25T19:09:10.141 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-25T19:09:10.145 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-25T19:09:10.145 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-25T19:09:10.191 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-25T19:09:10.191 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-25T19:09:10.419 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-25T19:09:10.524 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:09:10.942 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:09:11.207 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:09:11.207 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:09:11.207 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-25T19:09:11.207 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:09:11.207 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:09:11.207 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T19:09:11.404 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T19:09:11.978 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 1:22:53 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T19:09:12.133 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T19:09:12.133 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T19:09:12.133 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-03-25T19:09:12.133 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T19:09:12.134 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-25T19:09:12.134 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T19:09:12.134 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:09:12.134 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-25T19:09:12.134 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T19:09:12.134 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-03-25T19:09:12.134 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:09:12.135 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 337 k 2024-03-25T19:09:12.135 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 1.0 M 2024-03-25T19:09:12.135 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-25T19:09:13.535 INFO:teuthology.orchestra.run.smithi006.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 241 kB/s | 337 kB 00:01 2024-03-25T19:09:13.535 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:09:13.536 INFO:teuthology.orchestra.run.smithi006.stdout:Total 241 kB/s | 337 kB 00:01 2024-03-25T19:09:13.536 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-25T19:09:13.540 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-25T19:09:13.540 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-25T19:09:13.582 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-25T19:09:13.582 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-25T19:09:13.725 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-25T19:09:13.845 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:09:14.220 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:09:14.491 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:09:14.491 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:09:14.491 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-25T19:09:14.491 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:09:14.491 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:09:14.491 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T19:09:14.686 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T19:09:15.259 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 1:22:57 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T19:09:15.415 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T19:09:15.416 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T19:09:15.416 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-03-25T19:09:15.416 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T19:09:15.416 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-25T19:09:15.416 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T19:09:15.416 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:09:15.416 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-25T19:09:15.416 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T19:09:15.416 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-03-25T19:09:15.417 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:09:15.417 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 509 k 2024-03-25T19:09:15.417 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 2.5 M 2024-03-25T19:09:15.417 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-25T19:09:17.088 INFO:teuthology.orchestra.run.smithi006.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 305 kB/s | 509 kB 00:01 2024-03-25T19:09:17.089 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:09:17.089 INFO:teuthology.orchestra.run.smithi006.stdout:Total 304 kB/s | 509 kB 00:01 2024-03-25T19:09:17.089 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-25T19:09:17.093 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-25T19:09:17.093 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-25T19:09:17.135 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-25T19:09:17.136 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-25T19:09:17.306 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-25T19:09:17.397 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:09:17.790 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:09:18.090 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:09:18.090 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:09:18.090 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-25T19:09:18.090 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:09:18.090 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:09:18.090 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T19:09:18.286 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T19:09:18.862 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 1:23:00 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T19:09:19.017 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T19:09:19.017 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T19:09:19.017 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-03-25T19:09:19.017 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T19:09:19.018 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-25T19:09:19.018 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:09:19.018 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:09:19.018 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-25T19:09:19.018 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T19:09:19.018 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-03-25T19:09:19.018 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:09:19.018 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 52 M 2024-03-25T19:09:19.018 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 298 M 2024-03-25T19:09:19.018 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-25T19:10:14.975 INFO:teuthology.orchestra.run.smithi192.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 428 kB/s | 52 MB 02:04 2024-03-25T19:10:14.975 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:10:14.975 INFO:teuthology.orchestra.run.smithi192.stdout:Total 428 kB/s | 52 MB 02:04 2024-03-25T19:10:14.976 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-25T19:10:14.979 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-25T19:10:14.979 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-25T19:10:15.322 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-25T19:10:15.323 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-25T19:10:21.301 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-25T19:10:21.390 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:10:23.590 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:10:23.856 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:10:23.856 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:10:23.856 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-25T19:10:23.856 INFO:teuthology.orchestra.run.smithi192.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:10:23.856 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:10:23.856 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T19:10:24.057 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T19:10:24.618 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 1:24:09 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T19:10:24.769 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T19:10:24.769 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T19:10:24.769 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-03-25T19:10:24.770 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T19:10:24.770 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-25T19:10:24.770 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T19:10:24.770 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:10:24.770 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-25T19:10:24.770 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T19:10:24.770 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-03-25T19:10:24.770 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:10:24.770 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 615 k 2024-03-25T19:10:24.770 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 2.1 M 2024-03-25T19:10:24.770 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-25T19:10:27.360 INFO:teuthology.orchestra.run.smithi192.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 237 kB/s | 615 kB 00:02 2024-03-25T19:10:27.361 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:10:27.361 INFO:teuthology.orchestra.run.smithi192.stdout:Total 237 kB/s | 615 kB 00:02 2024-03-25T19:10:27.362 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-25T19:10:27.366 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-25T19:10:27.366 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-25T19:10:27.407 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-25T19:10:27.408 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-25T19:10:27.578 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-25T19:10:27.677 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:10:28.077 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:10:28.339 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:10:28.339 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:10:28.339 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-25T19:10:28.339 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:10:28.339 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:10:28.340 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T19:10:28.553 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-rados-debuginfo 2024-03-25T19:10:29.115 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 1:24:14 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T19:10:29.266 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T19:10:29.267 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T19:10:29.267 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-03-25T19:10:29.267 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T19:10:29.267 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-25T19:10:29.267 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T19:10:29.267 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:10:29.267 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-25T19:10:29.267 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T19:10:29.268 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-03-25T19:10:29.268 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:10:29.268 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 771 k 2024-03-25T19:10:29.268 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 2.7 M 2024-03-25T19:10:29.268 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-25T19:10:34.763 INFO:teuthology.orchestra.run.smithi192.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 140 kB/s | 771 kB 00:05 2024-03-25T19:10:34.764 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:10:34.764 INFO:teuthology.orchestra.run.smithi192.stdout:Total 140 kB/s | 771 kB 00:05 2024-03-25T19:10:34.764 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-25T19:10:34.768 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-25T19:10:34.768 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-25T19:10:34.811 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-25T19:10:34.811 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-25T19:10:34.994 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-25T19:10:35.095 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:10:35.504 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:10:35.762 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:10:35.762 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:10:35.762 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-25T19:10:35.762 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:10:35.762 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:10:35.762 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T19:10:36.012 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T19:10:36.573 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 1:24:21 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T19:10:36.722 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T19:10:36.723 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T19:10:36.723 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-03-25T19:10:36.723 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T19:10:36.724 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-25T19:10:36.724 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T19:10:36.724 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:10:36.724 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-25T19:10:36.724 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T19:10:36.724 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-03-25T19:10:36.724 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:10:36.724 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 1.1 M 2024-03-25T19:10:36.724 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 4.3 M 2024-03-25T19:10:36.724 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-25T19:10:43.174 INFO:teuthology.orchestra.run.smithi192.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 178 kB/s | 1.1 MB 00:06 2024-03-25T19:10:43.175 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:10:43.175 INFO:teuthology.orchestra.run.smithi192.stdout:Total 178 kB/s | 1.1 MB 00:06 2024-03-25T19:10:43.176 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-25T19:10:43.180 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-25T19:10:43.180 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-25T19:10:43.222 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-25T19:10:43.223 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-25T19:10:43.461 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-25T19:10:43.565 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:10:44.020 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:10:44.278 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:10:44.297 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:10:44.297 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-25T19:10:44.297 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:10:44.297 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:10:44.298 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T19:10:44.515 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T19:10:45.078 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 1:24:30 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T19:10:45.226 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T19:10:45.226 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T19:10:45.226 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-03-25T19:10:45.226 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T19:10:45.226 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-25T19:10:45.227 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T19:10:45.227 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:10:45.227 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-25T19:10:45.227 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T19:10:45.227 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-03-25T19:10:45.227 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:10:45.227 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 337 k 2024-03-25T19:10:45.227 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 1.0 M 2024-03-25T19:10:45.228 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-25T19:10:47.061 INFO:teuthology.orchestra.run.smithi192.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 184 kB/s | 337 kB 00:01 2024-03-25T19:10:47.061 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:10:47.062 INFO:teuthology.orchestra.run.smithi192.stdout:Total 184 kB/s | 337 kB 00:01 2024-03-25T19:10:47.062 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-25T19:10:47.066 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-25T19:10:47.066 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-25T19:10:47.106 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-25T19:10:47.106 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-25T19:10:47.247 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-25T19:10:47.343 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:10:47.692 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:10:47.951 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:10:47.951 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:10:47.952 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-25T19:10:47.952 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:10:47.952 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:10:47.952 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T19:10:48.175 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T19:10:48.733 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 1:24:33 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T19:10:48.882 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T19:10:48.883 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T19:10:48.883 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-03-25T19:10:48.883 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T19:10:48.884 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-25T19:10:48.884 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T19:10:48.884 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:10:48.884 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-25T19:10:48.884 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T19:10:48.884 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-03-25T19:10:48.884 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:10:48.884 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 509 k 2024-03-25T19:10:48.884 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 2.5 M 2024-03-25T19:10:48.884 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-25T19:10:51.690 INFO:teuthology.orchestra.run.smithi192.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 181 kB/s | 509 kB 00:02 2024-03-25T19:10:51.690 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:10:51.691 INFO:teuthology.orchestra.run.smithi192.stdout:Total 181 kB/s | 509 kB 00:02 2024-03-25T19:10:51.691 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-25T19:10:51.695 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-25T19:10:51.695 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-25T19:10:51.735 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-25T19:10:51.736 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-25T19:10:51.893 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-25T19:10:51.988 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:10:52.400 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:10:52.659 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:10:52.659 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:10:52.659 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-25T19:10:52.659 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:10:52.659 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:10:52.659 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T19:10:52.879 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T19:10:53.448 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 1:24:38 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T19:10:53.598 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T19:10:53.599 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T19:10:53.599 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-03-25T19:10:53.599 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T19:10:53.599 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-25T19:10:53.599 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:10:53.599 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:10:53.599 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-25T19:10:53.599 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T19:10:53.599 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-03-25T19:10:53.599 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:10:53.599 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 52 M 2024-03-25T19:10:53.599 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 298 M 2024-03-25T19:10:53.599 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-25T19:11:07.856 INFO:teuthology.orchestra.run.smithi006.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 485 kB/s | 52 MB 01:48 2024-03-25T19:11:07.857 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:11:07.857 INFO:teuthology.orchestra.run.smithi006.stdout:Total 485 kB/s | 52 MB 01:48 2024-03-25T19:11:07.857 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-25T19:11:07.861 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-25T19:11:07.861 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-25T19:11:08.211 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-25T19:11:08.211 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-25T19:11:14.470 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-25T19:11:14.566 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:11:16.636 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:11:16.912 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:11:16.913 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:11:16.913 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-25T19:11:16.913 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:11:16.913 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:11:16.913 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T19:11:17.146 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T19:11:17.719 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 1:24:59 ago on Mon 25 Mar 2024 05:46:18 PM UTC. 2024-03-25T19:11:17.875 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-25T19:11:17.876 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T19:11:17.876 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-03-25T19:11:17.876 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T19:11:17.876 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-25T19:11:17.876 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T19:11:17.876 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:11:17.876 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-25T19:11:17.876 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-25T19:11:17.876 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-03-25T19:11:17.876 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:11:17.877 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 1.3 M 2024-03-25T19:11:17.877 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 5.5 M 2024-03-25T19:11:17.877 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-25T19:11:20.625 INFO:teuthology.orchestra.run.smithi006.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 467 kB/s | 1.3 MB 00:02 2024-03-25T19:11:20.625 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:11:20.626 INFO:teuthology.orchestra.run.smithi006.stdout:Total 466 kB/s | 1.3 MB 00:02 2024-03-25T19:11:20.626 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-25T19:11:20.630 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-25T19:11:20.630 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-25T19:11:20.677 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-25T19:11:20.678 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-25T19:11:20.926 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-25T19:11:21.027 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:11:21.451 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:11:21.726 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:11:21.726 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:11:21.726 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-25T19:11:21.726 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:11:21.726 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:11:21.726 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-25T19:11:21.922 DEBUG:teuthology.parallel:result is None 2024-03-25T19:12:53.054 INFO:teuthology.orchestra.run.smithi192.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 442 kB/s | 52 MB 01:59 2024-03-25T19:12:53.055 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:12:53.055 INFO:teuthology.orchestra.run.smithi192.stdout:Total 442 kB/s | 52 MB 01:59 2024-03-25T19:12:53.055 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-25T19:12:53.058 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-25T19:12:53.058 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-25T19:12:53.392 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-25T19:12:53.392 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-25T19:12:59.287 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-25T19:12:59.393 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:13:01.393 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:13:01.657 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:13:01.658 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:13:01.658 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-25T19:13:01.658 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:13:01.658 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:13:01.658 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T19:13:01.893 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T19:13:02.459 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 1:26:47 ago on Mon 25 Mar 2024 05:46:15 PM UTC. 2024-03-25T19:13:02.610 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-25T19:13:02.611 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T19:13:02.611 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-03-25T19:13:02.611 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T19:13:02.611 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-25T19:13:02.611 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T19:13:02.611 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:13:02.611 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-25T19:13:02.611 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-25T19:13:02.611 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-03-25T19:13:02.611 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:13:02.611 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 1.3 M 2024-03-25T19:13:02.611 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 5.5 M 2024-03-25T19:13:02.612 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-25T19:13:04.989 INFO:teuthology.orchestra.run.smithi192.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 539 kB/s | 1.3 MB 00:02 2024-03-25T19:13:04.989 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:13:04.990 INFO:teuthology.orchestra.run.smithi192.stdout:Total 539 kB/s | 1.3 MB 00:02 2024-03-25T19:13:04.990 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-25T19:13:04.994 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-25T19:13:04.994 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-25T19:13:05.039 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-25T19:13:05.040 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-25T19:13:05.294 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-25T19:13:05.404 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:13:05.822 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:13:06.098 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:13:06.099 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:13:06.099 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-25T19:13:06.099 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:13:06.099 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:13:06.099 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-25T19:13:06.367 DEBUG:teuthology.parallel:result is None 2024-03-25T19:13:06.368 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T19:13:06.511 DEBUG:teuthology.orchestra.run.smithi006:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T19:13:06.536 INFO:teuthology.orchestra.run.smithi006.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T19:13:06.536 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T19:13:06.536 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T19:13:06.538 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T19:13:06.836 DEBUG:teuthology.orchestra.run.smithi192:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T19:13:06.857 INFO:teuthology.orchestra.run.smithi192.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T19:13:06.857 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T19:13:06.858 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T19:13:06.859 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T19:13:06.859 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-25T19:13:06.860 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T19:13:06.891 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-25T19:13:06.891 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T19:13:06.925 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T19:13:06.926 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-25T19:13:06.926 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T19:13:06.962 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T19:13:07.031 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-25T19:13:07.031 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T19:13:07.059 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T19:13:07.126 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T19:13:07.126 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-25T19:13:07.126 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T19:13:07.156 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T19:13:07.224 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-25T19:13:07.224 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T19:13:07.251 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T19:13:07.319 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T19:13:07.319 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-25T19:13:07.319 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T19:13:07.348 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T19:13:07.417 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-25T19:13:07.417 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T19:13:07.444 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T19:13:07.510 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T19:13:07.594 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T19:13:07.594 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 777 /var/log/ceph 2024-03-25T19:13:07.596 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 777 /var/log/ceph 2024-03-25T19:13:07.623 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T19:13:07.623 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T19:13:07.666 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T19:13:07.693 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T19:13:07.693 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T19:13:07.735 DEBUG:teuthology.orchestra.run.smithi192:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T19:13:07.791 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T19:13:07.791 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T19:13:07.791 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621303', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621303', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi006.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJzS5jxOxGYBkmXRwUv79FYTq0F6mAdD0VcTyIyO6APFUyBkiyxdulhH119XzUV6ATQR9i1XYNrHv24NcvH/MVc=', 'smithi192.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNSz2WXJW1fvNy/bJE8JQNd9/IgUiV02UxqYAAg6OOkO/eljBodBu+CP26wgr8RmCgIKY3bFpC1hVE40eVVCvdc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T19:13:07.791 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T19:13:07.804 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T19:13:07.819 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T19:13:07.860 DEBUG:teuthology.orchestra.run.smithi192:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T19:13:07.887 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-25T19:13:07.887 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T19:13:07.942 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T19:13:07.943 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_1 2024-03-25T19:13:07.997 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T19:13:07.998 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:13:07.998 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-25T19:13:07.998 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:13:07.998 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:13:07.998 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-03-25 17:52:47.143398756 +0000 2024-03-25T19:13:07.998 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-03-25 17:45:05.071559275 +0000 2024-03-25T19:13:07.998 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-03-25 17:45:05.071559275 +0000 2024-03-25T19:13:07.998 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-03-25 17:45:05.071559275 +0000 2024-03-25T19:13:07.998 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T19:13:08.064 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-03-25T19:13:08.064 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-03-25T19:13:08.064 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000190293 s, 2.7 MB/s 2024-03-25T19:13:08.065 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T19:13:08.122 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_2 2024-03-25T19:13:08.178 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T19:13:08.179 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:13:08.179 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-25T19:13:08.179 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:13:08.179 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:13:08.179 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-03-25 17:52:47.144398745 +0000 2024-03-25T19:13:08.179 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-03-25 17:45:05.418555104 +0000 2024-03-25T19:13:08.179 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-03-25 17:45:05.418555104 +0000 2024-03-25T19:13:08.179 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-03-25 17:45:05.418555104 +0000 2024-03-25T19:13:08.179 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T19:13:08.245 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-03-25T19:13:08.245 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-03-25T19:13:08.245 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000198403 s, 2.6 MB/s 2024-03-25T19:13:08.246 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T19:13:08.301 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_3 2024-03-25T19:13:08.357 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T19:13:08.357 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:13:08.357 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-25T19:13:08.357 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:13:08.357 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:13:08.357 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-03-25 17:52:47.144398745 +0000 2024-03-25T19:13:08.357 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-03-25 17:45:05.773550837 +0000 2024-03-25T19:13:08.357 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-03-25 17:45:05.773550837 +0000 2024-03-25T19:13:08.357 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-03-25 17:45:05.773550837 +0000 2024-03-25T19:13:08.357 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T19:13:08.422 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-03-25T19:13:08.422 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-03-25T19:13:08.423 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000179304 s, 2.9 MB/s 2024-03-25T19:13:08.424 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T19:13:08.479 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_4 2024-03-25T19:13:08.533 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T19:13:08.533 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:13:08.533 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-25T19:13:08.533 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:13:08.534 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:13:08.534 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-03-25 17:52:47.144398745 +0000 2024-03-25T19:13:08.534 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-03-25 17:45:06.117546703 +0000 2024-03-25T19:13:08.534 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-03-25 17:45:06.117546703 +0000 2024-03-25T19:13:08.534 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-03-25 17:45:06.117546703 +0000 2024-03-25T19:13:08.534 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T19:13:08.599 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-03-25T19:13:08.599 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-03-25T19:13:08.599 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000201954 s, 2.5 MB/s 2024-03-25T19:13:08.601 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T19:13:08.656 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:13:08.656 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-25T19:13:08.656 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T19:13:08.673 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T19:13:08.673 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_1 2024-03-25T19:13:08.728 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T19:13:08.728 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:13:08.728 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-25T19:13:08.729 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:13:08.729 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:13:08.729 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-03-25 17:52:08.083800293 +0000 2024-03-25T19:13:08.729 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-03-25 17:45:03.466743011 +0000 2024-03-25T19:13:08.729 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-03-25 17:45:03.466743011 +0000 2024-03-25T19:13:08.729 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-03-25 17:45:03.466743011 +0000 2024-03-25T19:13:08.729 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T19:13:08.796 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-03-25T19:13:08.796 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-03-25T19:13:08.796 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000194408 s, 2.6 MB/s 2024-03-25T19:13:08.797 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T19:13:08.853 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_2 2024-03-25T19:13:08.908 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T19:13:08.908 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:13:08.908 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-25T19:13:08.908 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:13:08.908 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:13:08.908 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-03-25 17:52:08.084800273 +0000 2024-03-25T19:13:08.908 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-03-25 17:45:03.810736144 +0000 2024-03-25T19:13:08.908 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-03-25 17:45:03.810736144 +0000 2024-03-25T19:13:08.908 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-03-25 17:45:03.810736144 +0000 2024-03-25T19:13:08.909 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T19:13:08.974 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-03-25T19:13:08.974 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-03-25T19:13:08.974 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000189468 s, 2.7 MB/s 2024-03-25T19:13:08.975 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T19:13:09.030 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_3 2024-03-25T19:13:09.085 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T19:13:09.086 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:13:09.086 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-25T19:13:09.086 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:13:09.086 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:13:09.086 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-03-25 17:52:08.084800273 +0000 2024-03-25T19:13:09.086 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-03-25 17:45:04.136729636 +0000 2024-03-25T19:13:09.086 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-03-25 17:45:04.136729636 +0000 2024-03-25T19:13:09.086 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-03-25 17:45:04.136729636 +0000 2024-03-25T19:13:09.086 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T19:13:09.151 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-03-25T19:13:09.151 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-03-25T19:13:09.151 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.00022476 s, 2.3 MB/s 2024-03-25T19:13:09.153 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T19:13:09.208 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_4 2024-03-25T19:13:09.262 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T19:13:09.262 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:13:09.262 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-25T19:13:09.262 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:13:09.262 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:13:09.263 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-03-25 17:52:08.085800253 +0000 2024-03-25T19:13:09.263 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-03-25 17:45:04.517722030 +0000 2024-03-25T19:13:09.263 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-03-25 17:45:04.517722030 +0000 2024-03-25T19:13:09.263 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-03-25 17:45:04.517722030 +0000 2024-03-25T19:13:09.263 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T19:13:09.327 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-03-25T19:13:09.327 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-03-25T19:13:09.327 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000198742 s, 2.6 MB/s 2024-03-25T19:13:09.328 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T19:13:09.383 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:13:09.383 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi006.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi192.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T19:13:09.383 INFO:tasks.ceph:Generating config... 2024-03-25T19:13:09.384 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T19:13:09.384 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T19:13:09.384 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T19:13:09.384 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T19:13:09.385 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T19:13:09.385 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T19:13:09.385 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T19:13:09.385 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T19:13:09.385 INFO:tasks.ceph:[global] ms type = async 2024-03-25T19:13:09.385 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T19:13:09.385 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T19:13:09.385 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T19:13:09.385 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T19:13:09.385 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T19:13:09.385 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T19:13:09.385 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T19:13:09.385 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T19:13:09.385 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T19:13:09.385 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T19:13:09.385 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T19:13:09.386 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T19:13:09.386 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T19:13:09.386 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T19:13:09.386 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T19:13:09.386 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T19:13:09.386 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T19:13:09.386 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T19:13:09.386 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T19:13:09.386 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T19:13:09.386 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T19:13:09.386 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T19:13:09.386 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T19:13:09.386 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T19:13:09.386 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T19:13:09.386 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T19:13:09.386 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T19:13:09.387 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T19:13:09.387 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T19:13:09.387 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T19:13:09.387 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T19:13:09.387 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T19:13:09.387 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T19:13:09.387 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T19:13:09.387 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T19:13:09.387 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T19:13:09.387 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T19:13:09.387 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T19:13:09.387 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T19:13:09.387 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T19:13:09.387 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T19:13:09.387 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T19:13:09.387 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T19:13:09.388 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T19:13:09.388 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T19:13:09.388 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T19:13:09.448 INFO:teuthology.orchestra.run.smithi006.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T19:13:09.451 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T19:13:09.497 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:13:09.525 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.6:6789]'), ('mon.c', '[v1:172.21.15.6:6790]'), ('mon.b', '[v1:172.21.15.192:6789]')] 2024-03-25T19:13:09.525 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.6:6789],[v1:172.21.15.6:6790],[v1:172.21.15.192:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T19:13:09.526 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-25T19:13:09.526 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T19:13:09.580 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.6:6789]' --addv c '[v1:172.21.15.6:6790]' --addv b '[v1:172.21.15.192:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:13:09.665 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:13:09.665 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: generated fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:13:09.665 INFO:teuthology.orchestra.run.smithi006.stdout:setting min_mon_release = quincy 2024-03-25T19:13:09.665 INFO:teuthology.orchestra.run.smithi006.stdout:epoch 0 2024-03-25T19:13:09.665 INFO:teuthology.orchestra.run.smithi006.stdout:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:13:09.665 INFO:teuthology.orchestra.run.smithi006.stdout:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:13:09.665 INFO:teuthology.orchestra.run.smithi006.stdout:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:13:09.665 INFO:teuthology.orchestra.run.smithi006.stdout:min_mon_release 17 (quincy) 2024-03-25T19:13:09.665 INFO:teuthology.orchestra.run.smithi006.stdout:election_strategy: 3 2024-03-25T19:13:09.666 INFO:teuthology.orchestra.run.smithi006.stdout:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:13:09.666 INFO:teuthology.orchestra.run.smithi006.stdout:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:13:09.666 INFO:teuthology.orchestra.run.smithi006.stdout:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:13:09.666 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T19:13:09.667 DEBUG:teuthology.orchestra.run.smithi006:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T19:13:09.681 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 673040aa-a547-4a0b-a8e8-76613d195b59... 2024-03-25T19:13:09.682 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T19:13:09.723 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T19:13:09.771 INFO:teuthology.orchestra.run.smithi006.stdout:[global] 2024-03-25T19:13:09.771 INFO:teuthology.orchestra.run.smithi006.stdout: chdir = "" 2024-03-25T19:13:09.771 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T19:13:09.771 INFO:teuthology.orchestra.run.smithi006.stdout: auth supported = cephx 2024-03-25T19:13:09.771 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:13:09.771 INFO:teuthology.orchestra.run.smithi006.stdout: filestore xattr use omap = true 2024-03-25T19:13:09.771 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:13:09.771 INFO:teuthology.orchestra.run.smithi006.stdout: mon clock drift allowed = 1.000 2024-03-25T19:13:09.771 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:13:09.771 INFO:teuthology.orchestra.run.smithi006.stdout: osd crush chooseleaf type = 0 2024-03-25T19:13:09.771 INFO:teuthology.orchestra.run.smithi006.stdout: auth debug = true 2024-03-25T19:13:09.771 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:13:09.771 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on old message = true 2024-03-25T19:13:09.771 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on bug = true 2024-03-25T19:13:09.772 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:13:09.772 INFO:teuthology.orchestra.run.smithi006.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T19:13:09.772 INFO:teuthology.orchestra.run.smithi006.stdout: mon pg warn max object skew = 0 2024-03-25T19:13:09.772 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:13:09.772 INFO:teuthology.orchestra.run.smithi006.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T19:13:09.772 INFO:teuthology.orchestra.run.smithi006.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T19:13:09.772 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:13:09.772 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default size = 2 2024-03-25T19:13:09.772 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:13:09.772 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow primary affinity = true 2024-03-25T19:13:09.772 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow pg remap = true 2024-03-25T19:13:09.772 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on legacy crush tunables = false 2024-03-25T19:13:09.772 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on crush straw calc version zero = false 2024-03-25T19:13:09.772 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on no sortbitwise = false 2024-03-25T19:13:09.772 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on osd down out interval zero = false 2024-03-25T19:13:09.773 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on too few osds = false 2024-03-25T19:13:09.773 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T19:13:09.773 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T19:13:09.773 INFO:teuthology.orchestra.run.smithi006.stdout: mon_allow_pool_size_one = true 2024-03-25T19:13:09.773 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:13:09.773 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T19:13:09.773 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:13:09.773 INFO:teuthology.orchestra.run.smithi006.stdout: osd default data pool replay window = 5 2024-03-25T19:13:09.773 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:13:09.773 INFO:teuthology.orchestra.run.smithi006.stdout: mon allow pool delete = true 2024-03-25T19:13:09.773 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:13:09.773 INFO:teuthology.orchestra.run.smithi006.stdout: mon cluster log file level = debug 2024-03-25T19:13:09.773 INFO:teuthology.orchestra.run.smithi006.stdout: debug asserts on shutdown = true 2024-03-25T19:13:09.773 INFO:teuthology.orchestra.run.smithi006.stdout: mon health detail to clog = false 2024-03-25T19:13:09.773 INFO:teuthology.orchestra.run.smithi006.stdout: mon host = "[v1:172.21.15.6:6789],[v1:172.21.15.6:6790],[v1:172.21.15.192:6789]" 2024-03-25T19:13:09.773 INFO:teuthology.orchestra.run.smithi006.stdout: mon client directed command retry = 5 2024-03-25T19:13:09.774 INFO:teuthology.orchestra.run.smithi006.stdout: mon election default strategy = 3 2024-03-25T19:13:09.774 INFO:teuthology.orchestra.run.smithi006.stdout: ms bind msgr2 = False 2024-03-25T19:13:09.774 INFO:teuthology.orchestra.run.smithi006.stdout: ms inject socket failures = 5000 2024-03-25T19:13:09.774 INFO:teuthology.orchestra.run.smithi006.stdout: ms type = async 2024-03-25T19:13:09.774 INFO:teuthology.orchestra.run.smithi006.stdout: osd heartbeat grace = 80 2024-03-25T19:13:09.774 INFO:teuthology.orchestra.run.smithi006.stdout: fsid = 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:13:09.774 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:13:09.774 INFO:teuthology.orchestra.run.smithi006.stdout:[osd] 2024-03-25T19:13:09.774 INFO:teuthology.orchestra.run.smithi006.stdout: osd journal size = 100 2024-03-25T19:13:09.774 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:13:09.774 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub load threshold = 5.0 2024-03-25T19:13:09.774 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub max interval = 600 2024-03-25T19:13:09.774 INFO:teuthology.orchestra.run.smithi006.stdout: osd mclock profile = high_recovery_ops 2024-03-25T19:13:09.774 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:13:09.775 INFO:teuthology.orchestra.run.smithi006.stdout: osd recover clone overlap = true 2024-03-25T19:13:09.775 INFO:teuthology.orchestra.run.smithi006.stdout: osd recovery max chunk = 1048576 2024-03-25T19:13:09.775 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:13:09.775 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug shutdown = true 2024-03-25T19:13:09.775 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug op order = true 2024-03-25T19:13:09.775 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify stray on activate = true 2024-03-25T19:13:09.775 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug trim objects = true 2024-03-25T19:13:09.775 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:13:09.775 INFO:teuthology.orchestra.run.smithi006.stdout: osd open classes on start = true 2024-03-25T19:13:09.775 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug pg log writeout = true 2024-03-25T19:13:09.775 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:13:09.775 INFO:teuthology.orchestra.run.smithi006.stdout: osd deep scrub update digest min age = 30 2024-03-25T19:13:09.775 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:13:09.775 INFO:teuthology.orchestra.run.smithi006.stdout: osd map max advance = 10 2024-03-25T19:13:09.775 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:13:09.775 INFO:teuthology.orchestra.run.smithi006.stdout: journal zero on create = true 2024-03-25T19:13:09.776 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:13:09.776 INFO:teuthology.orchestra.run.smithi006.stdout: filestore ondisk finisher threads = 3 2024-03-25T19:13:09.776 INFO:teuthology.orchestra.run.smithi006.stdout: filestore apply finisher threads = 3 2024-03-25T19:13:09.776 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:13:09.776 INFO:teuthology.orchestra.run.smithi006.stdout: bdev debug aio = true 2024-03-25T19:13:09.776 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug misdirected ops = true 2024-03-25T19:13:09.776 INFO:teuthology.orchestra.run.smithi006.stdout: bdev async discard = True 2024-03-25T19:13:09.776 INFO:teuthology.orchestra.run.smithi006.stdout: bdev enable discard = True 2024-03-25T19:13:09.776 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore allocator = stupid 2024-03-25T19:13:09.776 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore block size = 96636764160 2024-03-25T19:13:09.776 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore fsck on mount = True 2024-03-25T19:13:09.776 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore zero block detection = True 2024-03-25T19:13:09.776 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluefs = 1 2024-03-25T19:13:09.776 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluestore = 1 2024-03-25T19:13:09.776 INFO:teuthology.orchestra.run.smithi006.stdout: debug monc = 20 2024-03-25T19:13:09.777 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-25T19:13:09.777 INFO:teuthology.orchestra.run.smithi006.stdout: debug osd = 20 2024-03-25T19:13:09.777 INFO:teuthology.orchestra.run.smithi006.stdout: debug rocksdb = 10 2024-03-25T19:13:09.777 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T19:13:09.777 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd full ratio = 0.9 2024-03-25T19:13:09.777 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd nearfull ratio = 0.8 2024-03-25T19:13:09.777 INFO:teuthology.orchestra.run.smithi006.stdout: osd blocked scrub grace period = 3600 2024-03-25T19:13:09.777 INFO:teuthology.orchestra.run.smithi006.stdout: osd class default list = * 2024-03-25T19:13:09.777 INFO:teuthology.orchestra.run.smithi006.stdout: osd class load list = * 2024-03-25T19:13:09.777 INFO:teuthology.orchestra.run.smithi006.stdout: osd client watch timeout = 120 2024-03-25T19:13:09.777 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify cached snaps = True 2024-03-25T19:13:09.777 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify missing on start = True 2024-03-25T19:13:09.777 INFO:teuthology.orchestra.run.smithi006.stdout: osd failsafe full ratio = 0.95 2024-03-25T19:13:09.777 INFO:teuthology.orchestra.run.smithi006.stdout: osd fast shutdown = False 2024-03-25T19:13:09.777 INFO:teuthology.orchestra.run.smithi006.stdout: osd max markdown count = 1000 2024-03-25T19:13:09.778 INFO:teuthology.orchestra.run.smithi006.stdout: osd mclock override recovery settings = True 2024-03-25T19:13:09.778 INFO:teuthology.orchestra.run.smithi006.stdout: osd objectstore = bluestore 2024-03-25T19:13:09.778 INFO:teuthology.orchestra.run.smithi006.stdout: osd op queue = debug_random 2024-03-25T19:13:09.778 INFO:teuthology.orchestra.run.smithi006.stdout: osd op queue cut off = debug_random 2024-03-25T19:13:09.778 INFO:teuthology.orchestra.run.smithi006.stdout: osd shutdown pgref assert = True 2024-03-25T19:13:09.778 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:13:09.778 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr] 2024-03-25T19:13:09.778 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-25T19:13:09.778 INFO:teuthology.orchestra.run.smithi006.stdout: debug mgr = 20 2024-03-25T19:13:09.778 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-03-25T19:13:09.778 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-03-25T19:13:09.778 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:13:09.778 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:13:09.778 INFO:teuthology.orchestra.run.smithi006.stdout: mgr/telemetry/nag = false 2024-03-25T19:13:09.778 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:13:09.779 INFO:teuthology.orchestra.run.smithi006.stdout:[mon] 2024-03-25T19:13:09.779 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-25T19:13:09.779 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-03-25T19:13:09.779 INFO:teuthology.orchestra.run.smithi006.stdout: debug paxos = 20 2024-03-25T19:13:09.779 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-03-25T19:13:09.779 INFO:teuthology.orchestra.run.smithi006.stdout: mon data avail warn = 5 2024-03-25T19:13:09.779 INFO:teuthology.orchestra.run.smithi006.stdout: mon mgr mkfs grace = 240 2024-03-25T19:13:09.779 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:13:09.779 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd reporter subtree level = osd 2024-03-25T19:13:09.779 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd prime pg temp = true 2024-03-25T19:13:09.779 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:13:09.779 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:13:09.779 INFO:teuthology.orchestra.run.smithi006.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T19:13:09.779 INFO:teuthology.orchestra.run.smithi006.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T19:13:09.779 INFO:teuthology.orchestra.run.smithi006.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T19:13:09.779 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:13:09.780 INFO:teuthology.orchestra.run.smithi006.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T19:13:09.780 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T19:13:09.780 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T19:13:09.780 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:13:09.780 INFO:teuthology.orchestra.run.smithi006.stdout: # 1m isn't quite enough 2024-03-25T19:13:09.780 INFO:teuthology.orchestra.run.smithi006.stdout: mon_down_mkfs_grace = 2m 2024-03-25T19:13:09.780 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:13:09.780 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_filestore_osds = false 2024-03-25T19:13:09.780 INFO:teuthology.orchestra.run.smithi006.stdout: mon min osdmap epochs = 50 2024-03-25T19:13:09.780 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd crush smoke test = False 2024-03-25T19:13:09.780 INFO:teuthology.orchestra.run.smithi006.stdout: mon osdmap full prune interval = 2 2024-03-25T19:13:09.780 INFO:teuthology.orchestra.run.smithi006.stdout: mon osdmap full prune min = 15 2024-03-25T19:13:09.780 INFO:teuthology.orchestra.run.smithi006.stdout: mon osdmap full prune txsize = 2 2024-03-25T19:13:09.780 INFO:teuthology.orchestra.run.smithi006.stdout: mon scrub interval = 300 2024-03-25T19:13:09.780 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on pool no app = False 2024-03-25T19:13:09.780 INFO:teuthology.orchestra.run.smithi006.stdout: paxos service trim min = 10 2024-03-25T19:13:09.781 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:13:09.781 INFO:teuthology.orchestra.run.smithi006.stdout:[client] 2024-03-25T19:13:09.781 INFO:teuthology.orchestra.run.smithi006.stdout: rgw cache enabled = true 2024-03-25T19:13:09.781 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable ops log = true 2024-03-25T19:13:09.781 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable usage log = true 2024-03-25T19:13:09.781 INFO:teuthology.orchestra.run.smithi006.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T19:13:09.781 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T19:13:09.781 INFO:teuthology.orchestra.run.smithi006.stdout: debug monc = 20 2024-03-25T19:13:09.781 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-25T19:13:09.781 INFO:teuthology.orchestra.run.smithi006.stdout: debug objecter = 20 2024-03-25T19:13:09.781 INFO:teuthology.orchestra.run.smithi006.stdout: debug rados = 20 2024-03-25T19:13:09.781 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.a] 2024-03-25T19:13:09.781 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.c] 2024-03-25T19:13:09.781 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.b] 2024-03-25T19:13:09.782 INFO:teuthology.orchestra.run.smithi192.stdout:[global] 2024-03-25T19:13:09.782 INFO:teuthology.orchestra.run.smithi192.stdout: chdir = "" 2024-03-25T19:13:09.782 INFO:teuthology.orchestra.run.smithi192.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T19:13:09.782 INFO:teuthology.orchestra.run.smithi192.stdout: auth supported = cephx 2024-03-25T19:13:09.782 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:13:09.782 INFO:teuthology.orchestra.run.smithi192.stdout: filestore xattr use omap = true 2024-03-25T19:13:09.782 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:13:09.782 INFO:teuthology.orchestra.run.smithi192.stdout: mon clock drift allowed = 1.000 2024-03-25T19:13:09.782 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:13:09.782 INFO:teuthology.orchestra.run.smithi192.stdout: osd crush chooseleaf type = 0 2024-03-25T19:13:09.782 INFO:teuthology.orchestra.run.smithi192.stdout: auth debug = true 2024-03-25T19:13:09.783 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:13:09.783 INFO:teuthology.orchestra.run.smithi192.stdout: ms die on old message = true 2024-03-25T19:13:09.783 INFO:teuthology.orchestra.run.smithi192.stdout: ms die on bug = true 2024-03-25T19:13:09.783 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:13:09.783 INFO:teuthology.orchestra.run.smithi192.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T19:13:09.783 INFO:teuthology.orchestra.run.smithi192.stdout: mon pg warn max object skew = 0 2024-03-25T19:13:09.783 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:13:09.783 INFO:teuthology.orchestra.run.smithi192.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T19:13:09.783 INFO:teuthology.orchestra.run.smithi192.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T19:13:09.783 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:13:09.783 INFO:teuthology.orchestra.run.smithi192.stdout: osd pool default size = 2 2024-03-25T19:13:09.783 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:13:09.783 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd allow primary affinity = true 2024-03-25T19:13:09.783 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd allow pg remap = true 2024-03-25T19:13:09.783 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on legacy crush tunables = false 2024-03-25T19:13:09.784 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on crush straw calc version zero = false 2024-03-25T19:13:09.784 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on no sortbitwise = false 2024-03-25T19:13:09.784 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on osd down out interval zero = false 2024-03-25T19:13:09.784 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on too few osds = false 2024-03-25T19:13:09.784 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T19:13:09.784 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T19:13:09.784 INFO:teuthology.orchestra.run.smithi192.stdout: mon_allow_pool_size_one = true 2024-03-25T19:13:09.784 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:13:09.784 INFO:teuthology.orchestra.run.smithi192.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T19:13:09.784 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:13:09.784 INFO:teuthology.orchestra.run.smithi192.stdout: osd default data pool replay window = 5 2024-03-25T19:13:09.784 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:13:09.784 INFO:teuthology.orchestra.run.smithi192.stdout: mon allow pool delete = true 2024-03-25T19:13:09.784 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:13:09.784 INFO:teuthology.orchestra.run.smithi192.stdout: mon cluster log file level = debug 2024-03-25T19:13:09.784 INFO:teuthology.orchestra.run.smithi192.stdout: debug asserts on shutdown = true 2024-03-25T19:13:09.784 INFO:teuthology.orchestra.run.smithi192.stdout: mon health detail to clog = false 2024-03-25T19:13:09.785 INFO:teuthology.orchestra.run.smithi192.stdout: mon host = "[v1:172.21.15.6:6789],[v1:172.21.15.6:6790],[v1:172.21.15.192:6789]" 2024-03-25T19:13:09.785 INFO:teuthology.orchestra.run.smithi192.stdout: mon client directed command retry = 5 2024-03-25T19:13:09.785 INFO:teuthology.orchestra.run.smithi192.stdout: mon election default strategy = 3 2024-03-25T19:13:09.785 INFO:teuthology.orchestra.run.smithi192.stdout: ms bind msgr2 = False 2024-03-25T19:13:09.785 INFO:teuthology.orchestra.run.smithi192.stdout: ms inject socket failures = 5000 2024-03-25T19:13:09.785 INFO:teuthology.orchestra.run.smithi192.stdout: ms type = async 2024-03-25T19:13:09.785 INFO:teuthology.orchestra.run.smithi192.stdout: osd heartbeat grace = 80 2024-03-25T19:13:09.785 INFO:teuthology.orchestra.run.smithi192.stdout: fsid = 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:13:09.785 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:13:09.785 INFO:teuthology.orchestra.run.smithi192.stdout:[osd] 2024-03-25T19:13:09.785 INFO:teuthology.orchestra.run.smithi192.stdout: osd journal size = 100 2024-03-25T19:13:09.785 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:13:09.785 INFO:teuthology.orchestra.run.smithi192.stdout: osd scrub load threshold = 5.0 2024-03-25T19:13:09.785 INFO:teuthology.orchestra.run.smithi192.stdout: osd scrub max interval = 600 2024-03-25T19:13:09.785 INFO:teuthology.orchestra.run.smithi192.stdout: osd mclock profile = high_recovery_ops 2024-03-25T19:13:09.785 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:13:09.786 INFO:teuthology.orchestra.run.smithi192.stdout: osd recover clone overlap = true 2024-03-25T19:13:09.786 INFO:teuthology.orchestra.run.smithi192.stdout: osd recovery max chunk = 1048576 2024-03-25T19:13:09.786 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:13:09.786 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug shutdown = true 2024-03-25T19:13:09.786 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug op order = true 2024-03-25T19:13:09.786 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug verify stray on activate = true 2024-03-25T19:13:09.786 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug trim objects = true 2024-03-25T19:13:09.786 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:13:09.786 INFO:teuthology.orchestra.run.smithi192.stdout: osd open classes on start = true 2024-03-25T19:13:09.786 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug pg log writeout = true 2024-03-25T19:13:09.786 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:13:09.786 INFO:teuthology.orchestra.run.smithi192.stdout: osd deep scrub update digest min age = 30 2024-03-25T19:13:09.786 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:13:09.786 INFO:teuthology.orchestra.run.smithi192.stdout: osd map max advance = 10 2024-03-25T19:13:09.786 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:13:09.787 INFO:teuthology.orchestra.run.smithi192.stdout: journal zero on create = true 2024-03-25T19:13:09.787 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:13:09.787 INFO:teuthology.orchestra.run.smithi192.stdout: filestore ondisk finisher threads = 3 2024-03-25T19:13:09.787 INFO:teuthology.orchestra.run.smithi192.stdout: filestore apply finisher threads = 3 2024-03-25T19:13:09.787 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:13:09.787 INFO:teuthology.orchestra.run.smithi192.stdout: bdev debug aio = true 2024-03-25T19:13:09.787 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug misdirected ops = true 2024-03-25T19:13:09.787 INFO:teuthology.orchestra.run.smithi192.stdout: bdev async discard = True 2024-03-25T19:13:09.787 INFO:teuthology.orchestra.run.smithi192.stdout: bdev enable discard = True 2024-03-25T19:13:09.787 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore allocator = stupid 2024-03-25T19:13:09.787 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore block size = 96636764160 2024-03-25T19:13:09.787 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore fsck on mount = True 2024-03-25T19:13:09.787 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore zero block detection = True 2024-03-25T19:13:09.787 INFO:teuthology.orchestra.run.smithi192.stdout: debug bluefs = 1 2024-03-25T19:13:09.787 INFO:teuthology.orchestra.run.smithi192.stdout: debug bluestore = 1 2024-03-25T19:13:09.788 INFO:teuthology.orchestra.run.smithi192.stdout: debug monc = 20 2024-03-25T19:13:09.788 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2024-03-25T19:13:09.788 INFO:teuthology.orchestra.run.smithi192.stdout: debug osd = 20 2024-03-25T19:13:09.788 INFO:teuthology.orchestra.run.smithi192.stdout: debug rocksdb = 10 2024-03-25T19:13:09.788 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T19:13:09.788 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd full ratio = 0.9 2024-03-25T19:13:09.788 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd nearfull ratio = 0.8 2024-03-25T19:13:09.788 INFO:teuthology.orchestra.run.smithi192.stdout: osd blocked scrub grace period = 3600 2024-03-25T19:13:09.788 INFO:teuthology.orchestra.run.smithi192.stdout: osd class default list = * 2024-03-25T19:13:09.788 INFO:teuthology.orchestra.run.smithi192.stdout: osd class load list = * 2024-03-25T19:13:09.788 INFO:teuthology.orchestra.run.smithi192.stdout: osd client watch timeout = 120 2024-03-25T19:13:09.788 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug verify cached snaps = True 2024-03-25T19:13:09.788 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug verify missing on start = True 2024-03-25T19:13:09.788 INFO:teuthology.orchestra.run.smithi192.stdout: osd failsafe full ratio = 0.95 2024-03-25T19:13:09.788 INFO:teuthology.orchestra.run.smithi192.stdout: osd fast shutdown = False 2024-03-25T19:13:09.788 INFO:teuthology.orchestra.run.smithi192.stdout: osd max markdown count = 1000 2024-03-25T19:13:09.789 INFO:teuthology.orchestra.run.smithi192.stdout: osd mclock override recovery settings = True 2024-03-25T19:13:09.789 INFO:teuthology.orchestra.run.smithi192.stdout: osd objectstore = bluestore 2024-03-25T19:13:09.789 INFO:teuthology.orchestra.run.smithi192.stdout: osd op queue = debug_random 2024-03-25T19:13:09.789 INFO:teuthology.orchestra.run.smithi192.stdout: osd op queue cut off = debug_random 2024-03-25T19:13:09.789 INFO:teuthology.orchestra.run.smithi192.stdout: osd shutdown pgref assert = True 2024-03-25T19:13:09.789 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:13:09.789 INFO:teuthology.orchestra.run.smithi192.stdout:[mgr] 2024-03-25T19:13:09.789 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2024-03-25T19:13:09.789 INFO:teuthology.orchestra.run.smithi192.stdout: debug mgr = 20 2024-03-25T19:13:09.789 INFO:teuthology.orchestra.run.smithi192.stdout: debug mon = 20 2024-03-25T19:13:09.789 INFO:teuthology.orchestra.run.smithi192.stdout: debug auth = 20 2024-03-25T19:13:09.789 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:13:09.789 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:13:09.789 INFO:teuthology.orchestra.run.smithi192.stdout: mgr/telemetry/nag = false 2024-03-25T19:13:09.789 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:13:09.790 INFO:teuthology.orchestra.run.smithi192.stdout:[mon] 2024-03-25T19:13:09.790 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2024-03-25T19:13:09.790 INFO:teuthology.orchestra.run.smithi192.stdout: debug mon = 20 2024-03-25T19:13:09.790 INFO:teuthology.orchestra.run.smithi192.stdout: debug paxos = 20 2024-03-25T19:13:09.790 INFO:teuthology.orchestra.run.smithi192.stdout: debug auth = 20 2024-03-25T19:13:09.790 INFO:teuthology.orchestra.run.smithi192.stdout: mon data avail warn = 5 2024-03-25T19:13:09.790 INFO:teuthology.orchestra.run.smithi192.stdout: mon mgr mkfs grace = 240 2024-03-25T19:13:09.790 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:13:09.790 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd reporter subtree level = osd 2024-03-25T19:13:09.790 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd prime pg temp = true 2024-03-25T19:13:09.790 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:13:09.790 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:13:09.790 INFO:teuthology.orchestra.run.smithi192.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T19:13:09.790 INFO:teuthology.orchestra.run.smithi192.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T19:13:09.790 INFO:teuthology.orchestra.run.smithi192.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T19:13:09.791 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:13:09.791 INFO:teuthology.orchestra.run.smithi192.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T19:13:09.791 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T19:13:09.791 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T19:13:09.791 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:13:09.791 INFO:teuthology.orchestra.run.smithi192.stdout: # 1m isn't quite enough 2024-03-25T19:13:09.791 INFO:teuthology.orchestra.run.smithi192.stdout: mon_down_mkfs_grace = 2m 2024-03-25T19:13:09.791 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:13:09.791 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_filestore_osds = false 2024-03-25T19:13:09.791 INFO:teuthology.orchestra.run.smithi192.stdout: mon min osdmap epochs = 50 2024-03-25T19:13:09.791 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd crush smoke test = False 2024-03-25T19:13:09.791 INFO:teuthology.orchestra.run.smithi192.stdout: mon osdmap full prune interval = 2 2024-03-25T19:13:09.791 INFO:teuthology.orchestra.run.smithi192.stdout: mon osdmap full prune min = 15 2024-03-25T19:13:09.792 INFO:teuthology.orchestra.run.smithi192.stdout: mon osdmap full prune txsize = 2 2024-03-25T19:13:09.792 INFO:teuthology.orchestra.run.smithi192.stdout: mon scrub interval = 300 2024-03-25T19:13:09.792 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on pool no app = False 2024-03-25T19:13:09.792 INFO:teuthology.orchestra.run.smithi192.stdout: paxos service trim min = 10 2024-03-25T19:13:09.792 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-25T19:13:09.792 INFO:teuthology.orchestra.run.smithi192.stdout:[client] 2024-03-25T19:13:09.792 INFO:teuthology.orchestra.run.smithi192.stdout: rgw cache enabled = true 2024-03-25T19:13:09.792 INFO:teuthology.orchestra.run.smithi192.stdout: rgw enable ops log = true 2024-03-25T19:13:09.792 INFO:teuthology.orchestra.run.smithi192.stdout: rgw enable usage log = true 2024-03-25T19:13:09.792 INFO:teuthology.orchestra.run.smithi192.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T19:13:09.792 INFO:teuthology.orchestra.run.smithi192.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T19:13:09.792 INFO:teuthology.orchestra.run.smithi192.stdout: debug monc = 20 2024-03-25T19:13:09.792 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2024-03-25T19:13:09.792 INFO:teuthology.orchestra.run.smithi192.stdout: debug objecter = 20 2024-03-25T19:13:09.792 INFO:teuthology.orchestra.run.smithi192.stdout: debug rados = 20 2024-03-25T19:13:09.792 INFO:teuthology.orchestra.run.smithi192.stdout:[mon.a] 2024-03-25T19:13:09.793 INFO:teuthology.orchestra.run.smithi192.stdout:[mon.c] 2024-03-25T19:13:09.793 INFO:teuthology.orchestra.run.smithi192.stdout:[mon.b] 2024-03-25T19:13:09.794 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T19:13:09.794 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T19:13:09.849 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T19:13:09.849 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-25T19:13:09.849 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T19:13:09.863 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-25T19:13:09.863 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T19:13:09.917 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi006.front.sepia.ceph.com 2024-03-25T19:13:09.917 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-25T19:13:09.917 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T19:13:09.917 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:13:09.995 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-25T19:13:09.995 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:13:10.050 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi192.front.sepia.ceph.com 2024-03-25T19:13:10.050 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-25T19:13:10.050 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T19:13:10.050 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:13:10.090 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-25T19:13:10.090 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:13:10.145 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T19:13:10.145 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T19:13:10.145 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T19:13:10.199 INFO:teuthology.orchestra.run.smithi006.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T19:13:10.201 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T19:13:10.266 INFO:teuthology.orchestra.run.smithi192.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T19:13:10.268 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T19:13:10.269 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T19:13:10.269 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T19:13:10.310 INFO:teuthology.orchestra.run.smithi006.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T19:13:10.326 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T19:13:10.368 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T19:13:10.383 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T19:13:10.383 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi006.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi192.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T19:13:10.383 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T19:13:10.410 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:13:10.410 INFO:tasks.ceph:role: osd.0 2024-03-25T19:13:10.410 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi006.front.sepia.ceph.com 2024-03-25T19:13:10.410 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T19:13:10.483 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:13:10.483 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:13:10.483 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:13:10.483 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:13:10.483 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:13:10.483 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:13:10.483 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:13:10.484 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:13:10.484 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:13:10.484 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:13:10.985 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-03-25T19:13:10.987 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-03-25T19:13:10.987 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T19:13:11.118 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T19:13:11.152 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T19:13:11.219 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:13:11.219 INFO:tasks.ceph:role: osd.1 2024-03-25T19:13:11.219 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi006.front.sepia.ceph.com 2024-03-25T19:13:11.220 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T19:13:11.294 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:13:11.294 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:13:11.295 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:13:11.295 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:13:11.295 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:13:11.295 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:13:11.295 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:13:11.295 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:13:11.295 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:13:11.295 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:13:11.748 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-03-25T19:13:11.750 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-03-25T19:13:11.750 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T19:13:11.840 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T19:13:11.872 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T19:13:11.940 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:13:11.941 INFO:tasks.ceph:role: osd.2 2024-03-25T19:13:11.941 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi006.front.sepia.ceph.com 2024-03-25T19:13:11.941 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T19:13:12.016 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:13:12.016 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:13:12.016 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:13:12.016 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:13:12.016 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:13:12.016 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:13:12.016 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:13:12.016 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:13:12.016 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:13:12.016 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:13:12.500 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-03-25T19:13:12.502 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-03-25T19:13:12.502 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T19:13:12.658 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T19:13:12.692 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T19:13:12.761 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:13:12.761 INFO:tasks.ceph:role: osd.3 2024-03-25T19:13:12.761 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi006.front.sepia.ceph.com 2024-03-25T19:13:12.761 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T19:13:12.836 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:13:12.836 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:13:12.836 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:13:12.836 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:13:12.836 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:13:12.837 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:13:12.837 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:13:12.837 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:13:12.837 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:13:12.837 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:13:13.347 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-03-25T19:13:13.349 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-03-25T19:13:13.350 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T19:13:13.479 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T19:13:13.515 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:13:13.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:13.624+0000 7f488e212640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T19:13:13.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:13.625+0000 7f488e212640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T19:13:13.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:13.625+0000 7f488e212640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:13:13.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:13.625+0000 7f488e212640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:13:13.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:13.625+0000 7f488e212640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:13:13.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:13.625+0000 7f488e212640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T19:13:16.470 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T19:13:16.501 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:13:16.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:16.592+0000 7f07c25ac640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T19:13:16.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:16.592+0000 7f07c25ac640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T19:13:16.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:16.592+0000 7f07c25ac640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:13:16.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:16.592+0000 7f07c25ac640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:13:16.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:16.592+0000 7f07c25ac640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:13:16.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:16.593+0000 7f07c25ac640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T19:13:19.433 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T19:13:19.464 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:13:19.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:19.554+0000 7fa71f80d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T19:13:19.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:19.554+0000 7fa71f80d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T19:13:19.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:19.554+0000 7fa71f80d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:13:19.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:19.554+0000 7fa71f80d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:13:19.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:19.554+0000 7fa71f80d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:13:19.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:19.555+0000 7fa71f80d640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T19:13:22.396 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T19:13:22.427 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:13:22.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:22.518+0000 7fb5d9ef9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T19:13:22.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:22.518+0000 7fb5d9ef9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T19:13:22.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:22.518+0000 7fb5d9ef9640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:13:22.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:22.518+0000 7fb5d9ef9640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:13:22.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:22.518+0000 7fb5d9ef9640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:13:22.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:22.519+0000 7fb5d9ef9640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T19:13:25.354 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T19:13:25.385 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T19:13:25.417 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:13:25.417 INFO:tasks.ceph:role: osd.4 2024-03-25T19:13:25.417 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi192.front.sepia.ceph.com 2024-03-25T19:13:25.417 DEBUG:teuthology.orchestra.run.smithi192:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T19:13:25.495 INFO:teuthology.orchestra.run.smithi192.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:13:25.495 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:13:25.495 INFO:teuthology.orchestra.run.smithi192.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:13:25.495 INFO:teuthology.orchestra.run.smithi192.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:13:25.495 INFO:teuthology.orchestra.run.smithi192.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:13:25.495 INFO:teuthology.orchestra.run.smithi192.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:13:25.496 INFO:teuthology.orchestra.run.smithi192.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:13:25.496 INFO:teuthology.orchestra.run.smithi192.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:13:25.496 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:13:25.496 INFO:teuthology.orchestra.run.smithi192.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:13:26.015 INFO:teuthology.orchestra.run.smithi192.stdout:Discarding blocks...Done. 2024-03-25T19:13:26.017 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi192.front.sepia.ceph.com -o noatime 2024-03-25T19:13:26.017 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T19:13:26.170 DEBUG:teuthology.orchestra.run.smithi192:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T19:13:26.207 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T19:13:26.277 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:13:26.277 INFO:tasks.ceph:role: osd.5 2024-03-25T19:13:26.277 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi192.front.sepia.ceph.com 2024-03-25T19:13:26.277 DEBUG:teuthology.orchestra.run.smithi192:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T19:13:26.353 INFO:teuthology.orchestra.run.smithi192.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:13:26.354 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:13:26.354 INFO:teuthology.orchestra.run.smithi192.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:13:26.354 INFO:teuthology.orchestra.run.smithi192.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:13:26.354 INFO:teuthology.orchestra.run.smithi192.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:13:26.354 INFO:teuthology.orchestra.run.smithi192.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:13:26.354 INFO:teuthology.orchestra.run.smithi192.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:13:26.354 INFO:teuthology.orchestra.run.smithi192.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:13:26.354 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:13:26.354 INFO:teuthology.orchestra.run.smithi192.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:13:26.811 INFO:teuthology.orchestra.run.smithi192.stdout:Discarding blocks...Done. 2024-03-25T19:13:26.813 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi192.front.sepia.ceph.com -o noatime 2024-03-25T19:13:26.813 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T19:13:26.910 DEBUG:teuthology.orchestra.run.smithi192:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T19:13:26.944 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T19:13:27.012 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:13:27.012 INFO:tasks.ceph:role: osd.6 2024-03-25T19:13:27.012 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi192.front.sepia.ceph.com 2024-03-25T19:13:27.013 DEBUG:teuthology.orchestra.run.smithi192:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T19:13:27.088 INFO:teuthology.orchestra.run.smithi192.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:13:27.088 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:13:27.088 INFO:teuthology.orchestra.run.smithi192.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:13:27.088 INFO:teuthology.orchestra.run.smithi192.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:13:27.088 INFO:teuthology.orchestra.run.smithi192.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:13:27.089 INFO:teuthology.orchestra.run.smithi192.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:13:27.089 INFO:teuthology.orchestra.run.smithi192.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:13:27.089 INFO:teuthology.orchestra.run.smithi192.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:13:27.089 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:13:27.089 INFO:teuthology.orchestra.run.smithi192.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:13:27.577 INFO:teuthology.orchestra.run.smithi192.stdout:Discarding blocks...Done. 2024-03-25T19:13:27.580 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi192.front.sepia.ceph.com -o noatime 2024-03-25T19:13:27.580 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T19:13:27.719 DEBUG:teuthology.orchestra.run.smithi192:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T19:13:27.753 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T19:13:27.822 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:13:27.822 INFO:tasks.ceph:role: osd.7 2024-03-25T19:13:27.822 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi192.front.sepia.ceph.com 2024-03-25T19:13:27.822 DEBUG:teuthology.orchestra.run.smithi192:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T19:13:27.897 INFO:teuthology.orchestra.run.smithi192.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:13:27.897 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:13:27.897 INFO:teuthology.orchestra.run.smithi192.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:13:27.898 INFO:teuthology.orchestra.run.smithi192.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:13:27.898 INFO:teuthology.orchestra.run.smithi192.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:13:27.898 INFO:teuthology.orchestra.run.smithi192.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:13:27.898 INFO:teuthology.orchestra.run.smithi192.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:13:27.898 INFO:teuthology.orchestra.run.smithi192.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:13:27.898 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:13:27.898 INFO:teuthology.orchestra.run.smithi192.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:13:28.398 INFO:teuthology.orchestra.run.smithi192.stdout:Discarding blocks...Done. 2024-03-25T19:13:28.401 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi192.front.sepia.ceph.com -o noatime 2024-03-25T19:13:28.401 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T19:13:28.535 DEBUG:teuthology.orchestra.run.smithi192:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T19:13:28.568 DEBUG:teuthology.orchestra.run.smithi192:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:13:28.798 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:13:28.798+0000 7ffa88811640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T19:13:28.798 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:13:28.798+0000 7ffa88811640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T19:13:28.799 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:13:28.798+0000 7ffa88811640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:13:28.799 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:13:28.798+0000 7ffa88811640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:13:28.799 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:13:28.798+0000 7ffa88811640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:13:28.799 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:13:28.799+0000 7ffa88811640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T19:13:31.640 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T19:13:31.670 DEBUG:teuthology.orchestra.run.smithi192:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:13:31.758 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:13:31.757+0000 7fd8e3a48640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T19:13:31.758 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:13:31.758+0000 7fd8e3a48640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T19:13:31.758 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:13:31.758+0000 7fd8e3a48640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:13:31.758 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:13:31.758+0000 7fd8e3a48640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:13:31.758 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:13:31.758+0000 7fd8e3a48640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:13:31.759 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:13:31.758+0000 7fd8e3a48640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T19:13:34.595 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T19:13:34.624 DEBUG:teuthology.orchestra.run.smithi192:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:13:34.714 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:13:34.714+0000 7fc19558c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T19:13:34.714 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:13:34.714+0000 7fc19558c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T19:13:34.714 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:13:34.714+0000 7fc19558c640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:13:34.714 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:13:34.714+0000 7fc19558c640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:13:34.715 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:13:34.714+0000 7fc19558c640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:13:34.715 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:13:34.715+0000 7fc19558c640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T19:13:37.567 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T19:13:37.598 DEBUG:teuthology.orchestra.run.smithi192:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:13:37.690 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:13:37.689+0000 7f9cf1011640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T19:13:37.690 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:13:37.689+0000 7f9cf1011640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T19:13:37.690 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:13:37.689+0000 7f9cf1011640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:13:37.690 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:13:37.689+0000 7f9cf1011640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:13:37.690 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:13:37.689+0000 7f9cf1011640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:13:37.690 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:13:37.690+0000 7f9cf1011640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T19:13:40.533 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T19:13:40.564 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T19:13:40.564 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-25T19:13:40.564 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T19:13:40.595 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-25T19:13:40.596 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T19:13:40.666 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-25T19:13:40.666 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T19:13:40.736 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-25T19:13:40.736 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T19:13:40.805 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-25T19:13:40.805 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T19:13:40.873 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-25T19:13:40.874 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T19:13:40.905 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-25T19:13:40.905 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T19:13:40.976 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-25T19:13:40.976 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T19:13:41.044 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-25T19:13:41.045 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T19:13:41.113 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-25T19:13:41.113 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T19:13:41.181 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-25T19:13:41.181 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T19:13:41.196 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-25T19:13:41.196 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T19:13:41.237 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T19:13:41.237 DEBUG:teuthology.orchestra.run.smithi006:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T19:13:41.239 DEBUG:teuthology.orchestra.run.smithi192:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T19:13:41.280 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr.y] 2024-03-25T19:13:41.280 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDGzAFmQ0bwCxAAuWcBjR+JFZcsiyXub4RhDg== 2024-03-25T19:13:41.281 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.0] 2024-03-25T19:13:41.281 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDJzAFmnUFMJRAA1RGXfBLOUWuAa1G2JkUYbQ== 2024-03-25T19:13:41.281 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.1] 2024-03-25T19:13:41.281 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDMzAFmAQ9gIxAAP/B/9ttSkim6gayPQz85XQ== 2024-03-25T19:13:41.281 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.2] 2024-03-25T19:13:41.281 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDPzAFmD7cdIRAAe7MDxS/hTIBOUl1j9AAU7g== 2024-03-25T19:13:41.281 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.3] 2024-03-25T19:13:41.281 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDSzAFmahf4HhAAgTWqyOtFspGiDtz/uoL7zA== 2024-03-25T19:13:41.281 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr.x] 2024-03-25T19:13:41.281 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDGzAFmPbHwDxAAbqpH745Qf8vLSMvfThuVGw== 2024-03-25T19:13:41.281 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.4] 2024-03-25T19:13:41.282 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDYzAFm4LajLxAAyezg1L7H2VXmibX3WPMAug== 2024-03-25T19:13:41.282 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.5] 2024-03-25T19:13:41.282 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDbzAFmUKQ8LRAAt6EUTn7d7vXjwIccuNUl/A== 2024-03-25T19:13:41.282 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.6] 2024-03-25T19:13:41.282 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDezAFm5WmhKhAAy1+4yqAYMqLNXGCF6O5E4Q== 2024-03-25T19:13:41.282 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.7] 2024-03-25T19:13:41.282 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDhzAFmGuwnKRAARSiwhnjR01Jy7D3zDJrMRw== 2024-03-25T19:13:41.282 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-03-25T19:13:41.282 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDGzAFmGGCZEhAAsjVYnds7mdYjunt2JEZuEQ== 2024-03-25T19:13:41.282 INFO:teuthology.orchestra.run.smithi006.stdout:[client.1] 2024-03-25T19:13:41.282 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDGzAFml/MBFhAAbRL0bItC9r65zzt+tK1rOg== 2024-03-25T19:13:41.302 INFO:teuthology.orchestra.run.smithi192.stdout:[mgr.y] 2024-03-25T19:13:41.302 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDGzAFmQ0bwCxAAuWcBjR+JFZcsiyXub4RhDg== 2024-03-25T19:13:41.303 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.0] 2024-03-25T19:13:41.303 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDJzAFmnUFMJRAA1RGXfBLOUWuAa1G2JkUYbQ== 2024-03-25T19:13:41.303 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.1] 2024-03-25T19:13:41.303 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDMzAFmAQ9gIxAAP/B/9ttSkim6gayPQz85XQ== 2024-03-25T19:13:41.303 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.2] 2024-03-25T19:13:41.303 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDPzAFmD7cdIRAAe7MDxS/hTIBOUl1j9AAU7g== 2024-03-25T19:13:41.303 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.3] 2024-03-25T19:13:41.303 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDSzAFmahf4HhAAgTWqyOtFspGiDtz/uoL7zA== 2024-03-25T19:13:41.303 INFO:teuthology.orchestra.run.smithi192.stdout:[mgr.x] 2024-03-25T19:13:41.303 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDGzAFmPbHwDxAAbqpH745Qf8vLSMvfThuVGw== 2024-03-25T19:13:41.303 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.4] 2024-03-25T19:13:41.303 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDYzAFm4LajLxAAyezg1L7H2VXmibX3WPMAug== 2024-03-25T19:13:41.303 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.5] 2024-03-25T19:13:41.303 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDbzAFmUKQ8LRAAt6EUTn7d7vXjwIccuNUl/A== 2024-03-25T19:13:41.303 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.6] 2024-03-25T19:13:41.304 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDezAFm5WmhKhAAy1+4yqAYMqLNXGCF6O5E4Q== 2024-03-25T19:13:41.304 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.7] 2024-03-25T19:13:41.304 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDhzAFmGuwnKRAARSiwhnjR01Jy7D3zDJrMRw== 2024-03-25T19:13:41.304 INFO:teuthology.orchestra.run.smithi192.stdout:[client.0] 2024-03-25T19:13:41.304 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDGzAFmGGCZEhAAsjVYnds7mdYjunt2JEZuEQ== 2024-03-25T19:13:41.304 INFO:teuthology.orchestra.run.smithi192.stdout:[client.1] 2024-03-25T19:13:41.304 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDGzAFml/MBFhAAbRL0bItC9r65zzt+tK1rOg== 2024-03-25T19:13:41.305 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:13:41.326 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:13:41.397 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:13:41.399 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:13:41.451 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:13:41.453 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:13:41.504 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:13:41.506 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:13:41.557 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:13:41.559 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:13:41.610 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:13:41.612 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:13:41.663 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:13:41.665 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:13:41.717 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:13:41.719 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:13:41.770 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:13:41.772 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:13:41.824 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:13:41.826 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:13:41.878 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:13:41.880 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:13:41.935 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:13:41.937 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:13:41.991 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T19:13:41.991 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T19:13:42.017 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:13:42.113 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T19:13:42.139 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T19:13:42.205 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:13:42.302 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T19:13:42.328 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T19:13:42.357 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:13:42.454 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T19:13:42.481 DEBUG:teuthology.orchestra.run.smithi006:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:13:42.483 DEBUG:teuthology.orchestra.run.smithi192:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:13:42.535 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T19:13:42.535 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:13:42.535 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T19:13:42.536 DEBUG:teuthology.orchestra.run.smithi006:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T19:13:42.538 INFO:tasks.ceph.mon.a:Started 2024-03-25T19:13:42.538 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:13:42.538 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T19:13:42.538 DEBUG:teuthology.orchestra.run.smithi006:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T19:13:42.540 INFO:tasks.ceph.mon.c:Started 2024-03-25T19:13:42.540 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:13:42.540 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T19:13:42.540 DEBUG:teuthology.orchestra.run.smithi192:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T19:13:42.577 INFO:tasks.ceph.mon.b:Started 2024-03-25T19:13:42.577 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T19:13:42.577 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T19:13:42.578 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T19:13:42.580 INFO:tasks.ceph.mgr.y:Started 2024-03-25T19:13:42.580 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T19:13:42.580 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T19:13:42.582 INFO:tasks.ceph.mgr.x:Started 2024-03-25T19:13:42.583 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T19:13:42.583 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T19:13:42.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:42.685+0000 7faea09cb640 10 monclient: get_monmap_and_config 2024-03-25T19:13:42.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:42.685+0000 7faea09cb640 10 monclient: build_initial_monmap 2024-03-25T19:13:42.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:42.685+0000 7faea09cb640 10 monclient: monmap: 2024-03-25T19:13:42.685 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-03-25T19:13:42.685 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:13:42.685 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-03-25T19:13:42.686508+0000 2024-03-25T19:13:42.686 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-03-25T19:13:42.686508+0000 2024-03-25T19:13:42.686 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:13:42.686 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-03-25T19:13:42.686 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:13:42.686 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:13:42.686 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:13:42.686 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-03-25T19:13:42.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:42.686+0000 7faea09cb640 1 Processor -- start 2024-03-25T19:13:42.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:42.686+0000 7faea09cb640 1 -- start start 2024-03-25T19:13:42.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:42.686+0000 7faea09cb640 10 monclient: init 2024-03-25T19:13:42.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:42.686+0000 7faea09cb640 10 monclient: _reopen_session rank -1 2024-03-25T19:13:42.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:42.686+0000 7faea09cb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:13:42.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:42.686+0000 7faea09cb640 10 monclient(hunting): picked mon.noname-b con 0x7fae9c0b4f00 addr v1:172.21.15.6:6790/0 2024-03-25T19:13:42.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:42.686+0000 7faea09cb640 10 monclient(hunting): picked mon.noname-c con 0x7fae9c0b4510 addr v1:172.21.15.192:6789/0 2024-03-25T19:13:42.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:42.686+0000 7faea09cb640 10 monclient(hunting): picked mon.noname-a con 0x7fae9c142ad0 addr v1:172.21.15.6:6789/0 2024-03-25T19:13:42.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:42.686+0000 7faea09cb640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae9c0a9d90 con 0x7fae9c142ad0 2024-03-25T19:13:42.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:42.686+0000 7faea09cb640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae9c0aa2b0 con 0x7fae9c0b4510 2024-03-25T19:13:42.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:42.686+0000 7faea09cb640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae9c13fc60 con 0x7fae9c0b4f00 2024-03-25T19:13:42.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:42.686+0000 7faea09cb640 10 monclient(hunting): _renew_subs 2024-03-25T19:13:42.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:42.686+0000 7faea09cb640 10 monclient(hunting): authenticate will time out at 5926.926730s 2024-03-25T19:13:42.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:42.686+0000 7fae9a59c640 1 -- >> v1:172.21.15.6:6790/0 conn(0x7fae9c0b4f00 legacy=0x7fae9c1423c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6790/0 2024-03-25T19:13:42.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:42.686+0000 7fae9ad9d640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7fae9c142ad0 legacy=0x7fae9c14dc80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-03-25T19:13:42.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:42.687+0000 7fae93fff640 1 -- >> v1:172.21.15.192:6789/0 conn(0x7fae9c0b4510 legacy=0x7fae9c13f550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.192:6789/0 2024-03-25T19:13:42.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:42.888+0000 7fae9a59c640 1 -- >> v1:172.21.15.6:6790/0 conn(0x7fae9c0b4f00 legacy=0x7fae9c1423c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6790/0 2024-03-25T19:13:42.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:42.888+0000 7fae9ad9d640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7fae9c142ad0 legacy=0x7fae9c14dc80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-03-25T19:13:42.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:42.888+0000 7fae93fff640 1 -- >> v1:172.21.15.192:6789/0 conn(0x7fae9c0b4510 legacy=0x7fae9c13f550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.192:6789/0 2024-03-25T19:13:43.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:43.289+0000 7fae9a59c640 1 -- >> v1:172.21.15.6:6790/0 conn(0x7fae9c0b4f00 legacy=0x7fae9c1423c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6790/0 2024-03-25T19:13:43.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:43.289+0000 7fae9ad9d640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7fae9c142ad0 legacy=0x7fae9c14dc80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-03-25T19:13:43.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:43.289+0000 7fae93fff640 1 -- >> v1:172.21.15.192:6789/0 conn(0x7fae9c0b4510 legacy=0x7fae9c13f550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.192:6789/0 2024-03-25T19:13:44.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:44.090+0000 7fae9a59c640 1 -- >> v1:172.21.15.6:6790/0 conn(0x7fae9c0b4f00 legacy=0x7fae9c1423c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6790/0 2024-03-25T19:13:44.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:44.090+0000 7fae9ad9d640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7fae9c142ad0 legacy=0x7fae9c14dc80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-03-25T19:13:44.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:44.090+0000 7fae93fff640 1 -- >> v1:172.21.15.192:6789/0 conn(0x7fae9c0b4510 legacy=0x7fae9c13f550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.192:6789/0 2024-03-25T19:13:45.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:45.686+0000 7fae98d99640 10 monclient(hunting): tick 2024-03-25T19:13:45.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:45.686+0000 7fae98d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:13:45.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:45.686+0000 7fae98d99640 1 monclient(hunting): continuing hunt 2024-03-25T19:13:45.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:45.686+0000 7fae98d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:13:45.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:45.686+0000 7fae98d99640 1 -- >> v1:172.21.15.6:6790/0 conn(0x7fae9c0b4f00 legacy=0x7fae9c1423c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:13:45.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:45.686+0000 7fae98d99640 1 -- >> v1:172.21.15.192:6789/0 conn(0x7fae9c0b4510 legacy=0x7fae9c13f550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:13:45.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:45.686+0000 7fae98d99640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7fae9c142ad0 legacy=0x7fae9c14dc80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:13:45.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:45.687+0000 7fae98d99640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:13:45.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:45.687+0000 7fae98d99640 10 monclient(hunting): picked mon.noname-b con 0x7fae9c0b4f00 addr v1:172.21.15.6:6790/0 2024-03-25T19:13:45.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:45.687+0000 7fae98d99640 10 monclient(hunting): picked mon.noname-a con 0x7fae9c142ad0 addr v1:172.21.15.6:6789/0 2024-03-25T19:13:45.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:45.687+0000 7fae98d99640 10 monclient(hunting): picked mon.noname-c con 0x7fae9c0b4510 addr v1:172.21.15.192:6789/0 2024-03-25T19:13:45.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:45.687+0000 7fae98d99640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae9c13fdf0 con 0x7fae9c142ad0 2024-03-25T19:13:45.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:45.687+0000 7fae98d99640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae9c0a9d90 con 0x7fae9c0b4510 2024-03-25T19:13:45.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:45.687+0000 7fae9a59c640 1 -- >> v1:172.21.15.6:6790/0 conn(0x7fae9c0b4f00 legacy=0x7fae7c0056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6790/0 2024-03-25T19:13:45.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:45.687+0000 7fae98d99640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae9c0aa2b0 con 0x7fae9c0b4f00 2024-03-25T19:13:45.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:45.687+0000 7fae98d99640 10 monclient(hunting): _renew_subs 2024-03-25T19:13:45.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:45.687+0000 7fae93fff640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7fae9c142ad0 legacy=0x7fae7c008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-03-25T19:13:45.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:45.687+0000 7fae9ad9d640 1 -- >> v1:172.21.15.192:6789/0 conn(0x7fae9c0b4510 legacy=0x7fae7c00c700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.192:6789/0 2024-03-25T19:13:45.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:45.888+0000 7fae9a59c640 1 -- >> v1:172.21.15.6:6790/0 conn(0x7fae9c0b4f00 legacy=0x7fae7c0056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6790/0 2024-03-25T19:13:45.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:45.888+0000 7fae93fff640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7fae9c142ad0 legacy=0x7fae7c008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-03-25T19:13:45.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:45.888+0000 7fae9ad9d640 1 -- >> v1:172.21.15.192:6789/0 conn(0x7fae9c0b4510 legacy=0x7fae7c00c700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.192:6789/0 2024-03-25T19:13:46.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:46.289+0000 7fae9a59c640 1 -- >> v1:172.21.15.6:6790/0 conn(0x7fae9c0b4f00 legacy=0x7fae7c0056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6790/0 2024-03-25T19:13:46.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:46.289+0000 7fae93fff640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7fae9c142ad0 legacy=0x7fae7c008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-03-25T19:13:46.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:46.289+0000 7fae9ad9d640 1 -- >> v1:172.21.15.192:6789/0 conn(0x7fae9c0b4510 legacy=0x7fae7c00c700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.192:6789/0 2024-03-25T19:13:47.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:47.090+0000 7fae93fff640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7fae9c142ad0 legacy=0x7fae7c008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-03-25T19:13:47.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:47.090+0000 7fae9a59c640 1 -- >> v1:172.21.15.6:6790/0 conn(0x7fae9c0b4f00 legacy=0x7fae7c0056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6790/0 2024-03-25T19:13:47.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:47.090+0000 7fae9ad9d640 1 -- >> v1:172.21.15.192:6789/0 conn(0x7fae9c0b4510 legacy=0x7fae7c00c700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.192:6789/0 2024-03-25T19:13:48.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:48.687+0000 7fae98d99640 10 monclient(hunting): tick 2024-03-25T19:13:48.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:48.687+0000 7fae98d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:13:48.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:48.687+0000 7fae98d99640 1 monclient(hunting): continuing hunt 2024-03-25T19:13:48.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:48.687+0000 7fae98d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:13:48.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:48.687+0000 7fae98d99640 1 -- >> v1:172.21.15.6:6790/0 conn(0x7fae9c0b4f00 legacy=0x7fae7c0056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:13:48.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:48.687+0000 7fae98d99640 1 -- >> v1:172.21.15.192:6789/0 conn(0x7fae9c0b4510 legacy=0x7fae7c00c700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:13:48.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:48.687+0000 7fae98d99640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7fae9c142ad0 legacy=0x7fae7c008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:13:48.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:48.687+0000 7fae98d99640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:13:48.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:48.687+0000 7fae98d99640 10 monclient(hunting): picked mon.noname-c con 0x7fae9c0b4510 addr v1:172.21.15.192:6789/0 2024-03-25T19:13:48.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:48.687+0000 7fae98d99640 10 monclient(hunting): picked mon.noname-a con 0x7fae9c142ad0 addr v1:172.21.15.6:6789/0 2024-03-25T19:13:48.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:48.687+0000 7fae98d99640 10 monclient(hunting): picked mon.noname-b con 0x7fae9c0b4f00 addr v1:172.21.15.6:6790/0 2024-03-25T19:13:48.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:48.687+0000 7fae98d99640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae9c13fc60 con 0x7fae9c142ad0 2024-03-25T19:13:48.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:48.687+0000 7fae98d99640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae9c13fdf0 con 0x7fae9c0b4510 2024-03-25T19:13:48.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:48.687+0000 7fae98d99640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae9c0a9d90 con 0x7fae9c0b4f00 2024-03-25T19:13:48.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:48.687+0000 7fae98d99640 10 monclient(hunting): _renew_subs 2024-03-25T19:13:48.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:48.687+0000 7fae9ad9d640 1 -- >> v1:172.21.15.6:6790/0 conn(0x7fae9c0b4f00 legacy=0x7fae7c0056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6790/0 2024-03-25T19:13:48.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:48.687+0000 7fae93fff640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7fae9c142ad0 legacy=0x7fae7c008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-03-25T19:13:48.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:48.687+0000 7fae9a59c640 1 -- >> v1:172.21.15.192:6789/0 conn(0x7fae9c0b4510 legacy=0x7fae7c00ff50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.192:6789/0 2024-03-25T19:13:48.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:48.888+0000 7fae93fff640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7fae9c142ad0 legacy=0x7fae7c008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-03-25T19:13:48.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:48.888+0000 7fae9ad9d640 1 -- >> v1:172.21.15.6:6790/0 conn(0x7fae9c0b4f00 legacy=0x7fae7c0056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6790/0 2024-03-25T19:13:48.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:48.888+0000 7fae9a59c640 1 -- >> v1:172.21.15.192:6789/0 conn(0x7fae9c0b4510 legacy=0x7fae7c00ff50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.192:6789/0 2024-03-25T19:13:49.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:49.289+0000 7fae93fff640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7fae9c142ad0 legacy=0x7fae7c008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-03-25T19:13:49.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:49.289+0000 7fae9ad9d640 1 -- >> v1:172.21.15.6:6790/0 conn(0x7fae9c0b4f00 legacy=0x7fae7c0056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6790/0 2024-03-25T19:13:49.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:49.289+0000 7fae9a59c640 1 -- >> v1:172.21.15.192:6789/0 conn(0x7fae9c0b4510 legacy=0x7fae7c00ff50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.192:6789/0 2024-03-25T19:13:49.541 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:00:06.859 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:13:49.847 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:00:07.165 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:13:50.020 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:00:07.366 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:13:50.024 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:00:07.370 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:13:50.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:50.090+0000 7fae9ad9d640 1 -- >> v1:172.21.15.6:6790/0 conn(0x7fae9c0b4f00 legacy=0x7fae7c0056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6790/0 2024-03-25T19:13:50.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:50.090+0000 7fae93fff640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7fae9c142ad0 legacy=0x7fae7c008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-03-25T19:13:50.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:50.090+0000 7fae9a59c640 1 -- >> v1:172.21.15.192:6789/0 conn(0x7fae9c0b4510 legacy=0x7fae7c00ff50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.192:6789/0 2024-03-25T19:13:50.347 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:00:07.694 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:13:50.350 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:00:07.697 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:13:51.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:51.687+0000 7fae98d99640 10 monclient(hunting): tick 2024-03-25T19:13:51.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:51.687+0000 7fae98d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:13:51.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:51.687+0000 7fae98d99640 1 monclient(hunting): continuing hunt 2024-03-25T19:13:51.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:51.687+0000 7fae98d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:13:51.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:51.687+0000 7fae98d99640 1 -- >> v1:172.21.15.6:6790/0 conn(0x7fae9c0b4f00 legacy=0x7fae7c0056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:13:51.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:51.687+0000 7fae98d99640 1 -- >> v1:172.21.15.192:6789/0 conn(0x7fae9c0b4510 legacy=0x7fae7c00ff50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:13:51.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:51.687+0000 7fae98d99640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7fae9c142ad0 legacy=0x7fae7c008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:13:51.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:51.687+0000 7fae98d99640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:13:51.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:51.687+0000 7fae98d99640 10 monclient(hunting): picked mon.noname-b con 0x7fae9c0b4f00 addr v1:172.21.15.6:6790/0 2024-03-25T19:13:51.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:51.687+0000 7fae98d99640 10 monclient(hunting): picked mon.noname-a con 0x7fae9c142ad0 addr v1:172.21.15.6:6789/0 2024-03-25T19:13:51.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:51.687+0000 7fae98d99640 10 monclient(hunting): picked mon.noname-c con 0x7fae9c0b4510 addr v1:172.21.15.192:6789/0 2024-03-25T19:13:51.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:51.687+0000 7fae98d99640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae9c0aa2b0 con 0x7fae9c142ad0 2024-03-25T19:13:51.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:51.687+0000 7fae98d99640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae9c13fc60 con 0x7fae9c0b4510 2024-03-25T19:13:51.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:51.687+0000 7fae98d99640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae9c13fdf0 con 0x7fae9c0b4f00 2024-03-25T19:13:51.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:51.687+0000 7fae98d99640 10 monclient(hunting): _renew_subs 2024-03-25T19:13:51.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:51.744+0000 7fae9ad9d640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fae9c0b4510 0x7fae7c00ff10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:39982/0 (socket says 172.21.15.6:39982) 2024-03-25T19:13:51.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:51.744+0000 7fae9ad9d640 1 -- 172.21.15.6:0/784594232 learned_addr learned my addr 172.21.15.6:0/784594232 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:13:52.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:52.040+0000 7fae93fff640 1 -- 172.21.15.6:0/784594232 >> v1:172.21.15.6:6789/0 conn(0x7fae9c142ad0 legacy=0x7fae7c0056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-25T19:13:52.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:52.040+0000 7fae93fff640 1 -- 172.21.15.6:0/784594232 >> v1:172.21.15.6:6789/0 conn(0x7fae9c142ad0 legacy=0x7fae7c0056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:13:52.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:52.040+0000 7fae93fff640 1 --1- 172.21.15.6:0/784594232 >> v1:172.21.15.6:6789/0 conn(0x7fae9c142ad0 0x7fae7c0056d0 :-1 s=OPENED pgs=2 cs=1 l=1).handle_message read tag failed 2024-03-25T19:13:52.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:52.040+0000 7fae93fff640 1 --1- 172.21.15.6:0/784594232 >> v1:172.21.15.6:6789/0 conn(0x7fae9c142ad0 0x7fae7c0056d0 :-1 s=OPENED pgs=2 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:13:52.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:52.040+0000 7fae99d9b640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.6:6789/0 2024-03-25T19:13:52.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:52.065+0000 7fae9a59c640 1 -- 172.21.15.6:0/784594232 >> v1:172.21.15.6:6790/0 conn(0x7fae9c0b4f00 legacy=0x7fae7c010660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-25T19:13:52.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:52.065+0000 7fae9a59c640 1 -- 172.21.15.6:0/784594232 >> v1:172.21.15.6:6790/0 conn(0x7fae9c0b4f00 legacy=0x7fae7c010660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:13:52.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:52.065+0000 7fae9a59c640 1 --1- 172.21.15.6:0/784594232 >> v1:172.21.15.6:6790/0 conn(0x7fae9c0b4f00 0x7fae7c010660 :-1 s=OPENED pgs=2 cs=1 l=1).handle_message read tag failed 2024-03-25T19:13:52.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:52.065+0000 7fae9a59c640 1 --1- 172.21.15.6:0/784594232 >> v1:172.21.15.6:6790/0 conn(0x7fae9c0b4f00 0x7fae7c010660 :-1 s=OPENED pgs=2 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:13:52.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:52.065+0000 7fae99d9b640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.6:6790/0 2024-03-25T19:13:54.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:54.688+0000 7fae98d99640 10 monclient(hunting): tick 2024-03-25T19:13:54.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:54.688+0000 7fae98d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:13:54.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:54.688+0000 7fae98d99640 1 monclient(hunting): continuing hunt 2024-03-25T19:13:54.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:54.688+0000 7fae98d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:13:54.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:54.688+0000 7fae98d99640 1 -- 172.21.15.6:0/784594232 >> v1:172.21.15.6:6790/0 conn(0x7fae9c0b4f00 legacy=0x7fae7c010660 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:13:54.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:54.688+0000 7fae98d99640 1 -- 172.21.15.6:0/784594232 >> v1:172.21.15.192:6789/0 conn(0x7fae9c0b4510 legacy=0x7fae7c00ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:13:54.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:54.688+0000 7fae98d99640 1 -- 172.21.15.6:0/784594232 >> v1:172.21.15.6:6789/0 conn(0x7fae9c142ad0 legacy=0x7fae7c0056d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:13:54.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:54.688+0000 7fae98d99640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:13:54.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:54.688+0000 7fae98d99640 10 monclient(hunting): picked mon.noname-b con 0x7fae9c0b4f00 addr v1:172.21.15.6:6790/0 2024-03-25T19:13:54.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:54.688+0000 7fae98d99640 10 monclient(hunting): picked mon.noname-c con 0x7fae7c00c790 addr v1:172.21.15.192:6789/0 2024-03-25T19:13:54.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:54.688+0000 7fae98d99640 10 monclient(hunting): picked mon.noname-a con 0x7fae9c142ad0 addr v1:172.21.15.6:6789/0 2024-03-25T19:13:54.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:54.688+0000 7fae98d99640 1 -- 172.21.15.6:0/784594232 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae9c0a9d90 con 0x7fae9c142ad0 2024-03-25T19:13:54.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:54.688+0000 7fae98d99640 1 -- 172.21.15.6:0/784594232 --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae7c00cc50 con 0x7fae7c00c790 2024-03-25T19:13:54.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:54.688+0000 7fae98d99640 1 -- 172.21.15.6:0/784594232 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae7c00ff60 con 0x7fae9c0b4f00 2024-03-25T19:13:54.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:54.688+0000 7fae98d99640 10 monclient(hunting): _renew_subs 2024-03-25T19:13:57.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:57.687+0000 7fae98d99640 10 monclient(hunting): tick 2024-03-25T19:13:57.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:57.687+0000 7fae98d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:13:57.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:57.687+0000 7fae98d99640 1 monclient(hunting): continuing hunt 2024-03-25T19:13:57.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:57.687+0000 7fae98d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:13:57.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:57.687+0000 7fae98d99640 1 -- 172.21.15.6:0/784594232 >> v1:172.21.15.6:6790/0 conn(0x7fae9c0b4f00 legacy=0x7fae7c010660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:13:57.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:57.688+0000 7fae98d99640 1 -- 172.21.15.6:0/784594232 >> v1:172.21.15.192:6789/0 conn(0x7fae7c00c790 legacy=0x7fae7c008e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:13:57.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:57.688+0000 7fae98d99640 1 -- 172.21.15.6:0/784594232 >> v1:172.21.15.6:6789/0 conn(0x7fae9c142ad0 legacy=0x7fae7c0056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:13:57.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:57.688+0000 7fae98d99640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:13:57.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:57.688+0000 7fae98d99640 10 monclient(hunting): picked mon.noname-b con 0x7fae7c00ce30 addr v1:172.21.15.6:6790/0 2024-03-25T19:13:57.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:57.688+0000 7fae98d99640 10 monclient(hunting): picked mon.noname-a con 0x7fae7c00d980 addr v1:172.21.15.6:6789/0 2024-03-25T19:13:57.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:57.688+0000 7fae98d99640 10 monclient(hunting): picked mon.noname-c con 0x7fae7c00c790 addr v1:172.21.15.192:6789/0 2024-03-25T19:13:57.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:57.688+0000 7fae98d99640 1 -- 172.21.15.6:0/784594232 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae7c010190 con 0x7fae7c00d980 2024-03-25T19:13:57.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:57.688+0000 7fae98d99640 1 -- 172.21.15.6:0/784594232 --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae7c006e70 con 0x7fae7c00c790 2024-03-25T19:13:57.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:57.688+0000 7fae98d99640 1 -- 172.21.15.6:0/784594232 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae7c008010 con 0x7fae7c00ce30 2024-03-25T19:13:57.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:13:57.688+0000 7fae98d99640 10 monclient(hunting): _renew_subs 2024-03-25T19:14:00.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:00.688+0000 7fae98d99640 10 monclient(hunting): tick 2024-03-25T19:14:00.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:00.688+0000 7fae98d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:14:00.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:00.688+0000 7fae98d99640 1 monclient(hunting): continuing hunt 2024-03-25T19:14:00.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:00.688+0000 7fae98d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:14:00.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:00.688+0000 7fae98d99640 1 -- 172.21.15.6:0/784594232 >> v1:172.21.15.6:6790/0 conn(0x7fae7c00ce30 legacy=0x7fae7c00d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:00.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:00.688+0000 7fae98d99640 1 -- 172.21.15.6:0/784594232 >> v1:172.21.15.192:6789/0 conn(0x7fae7c00c790 legacy=0x7fae7c010660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:00.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:00.688+0000 7fae98d99640 1 -- 172.21.15.6:0/784594232 >> v1:172.21.15.6:6789/0 conn(0x7fae7c00d980 legacy=0x7fae7c015050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:00.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:00.688+0000 7fae98d99640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:14:00.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:00.688+0000 7fae98d99640 10 monclient(hunting): picked mon.noname-a con 0x7fae7c007000 addr v1:172.21.15.6:6789/0 2024-03-25T19:14:00.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:00.688+0000 7fae98d99640 10 monclient(hunting): picked mon.noname-b con 0x7fae7c007af0 addr v1:172.21.15.6:6790/0 2024-03-25T19:14:00.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:00.688+0000 7fae98d99640 10 monclient(hunting): picked mon.noname-c con 0x7fae7c0066e0 addr v1:172.21.15.192:6789/0 2024-03-25T19:14:00.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:00.688+0000 7fae98d99640 1 -- 172.21.15.6:0/784594232 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae7c009200 con 0x7fae7c007000 2024-03-25T19:14:00.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:00.688+0000 7fae98d99640 1 -- 172.21.15.6:0/784594232 --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae7c006c00 con 0x7fae7c0066e0 2024-03-25T19:14:00.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:00.688+0000 7fae98d99640 1 -- 172.21.15.6:0/784594232 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae7c00d2c0 con 0x7fae7c007af0 2024-03-25T19:14:00.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:00.688+0000 7fae98d99640 10 monclient(hunting): _renew_subs 2024-03-25T19:14:03.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:03.689+0000 7fae98d99640 10 monclient(hunting): tick 2024-03-25T19:14:03.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:03.689+0000 7fae98d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:14:03.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:03.689+0000 7fae98d99640 1 monclient(hunting): continuing hunt 2024-03-25T19:14:03.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:03.689+0000 7fae98d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:14:03.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:03.689+0000 7fae98d99640 1 -- 172.21.15.6:0/784594232 >> v1:172.21.15.6:6790/0 conn(0x7fae7c007af0 legacy=0x7fae7c0081a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:03.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:03.689+0000 7fae98d99640 1 -- 172.21.15.6:0/784594232 >> v1:172.21.15.192:6789/0 conn(0x7fae7c0066e0 legacy=0x7fae7c010660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:03.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:03.689+0000 7fae98d99640 1 -- 172.21.15.6:0/784594232 >> v1:172.21.15.6:6789/0 conn(0x7fae7c007000 legacy=0x7fae7c0073e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:03.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:03.689+0000 7fae98d99640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:14:03.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:03.689+0000 7fae98d99640 10 monclient(hunting): picked mon.noname-a con 0x7fae7c0088b0 addr v1:172.21.15.6:6789/0 2024-03-25T19:14:03.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:03.689+0000 7fae98d99640 10 monclient(hunting): picked mon.noname-c con 0x7fae7c008cf0 addr v1:172.21.15.192:6789/0 2024-03-25T19:14:03.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:03.689+0000 7fae98d99640 10 monclient(hunting): picked mon.noname-b con 0x7fae7c007af0 addr v1:172.21.15.6:6790/0 2024-03-25T19:14:03.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:03.689+0000 7fae98d99640 1 -- 172.21.15.6:0/784594232 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae7c00d4f0 con 0x7fae7c0088b0 2024-03-25T19:14:03.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:03.689+0000 7fae98d99640 1 -- 172.21.15.6:0/784594232 --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae7c00d740 con 0x7fae7c008cf0 2024-03-25T19:14:03.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:03.689+0000 7fae98d99640 1 -- 172.21.15.6:0/784594232 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae7c0081a0 con 0x7fae7c007af0 2024-03-25T19:14:03.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:03.689+0000 7fae98d99640 10 monclient(hunting): _renew_subs 2024-03-25T19:14:05.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.884+0000 7fae99d9b640 1 -- 172.21.15.6:0/784594232 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 228081108 0 0) 0x7fae7c00d4f0 con 0x7fae7c0088b0 2024-03-25T19:14:05.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.884+0000 7fae99d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:14:05.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.884+0000 7fae99d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:05.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.884+0000 7fae99d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:05.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.884+0000 7fae99d9b640 10 monclient(hunting): my global_id is 4101 2024-03-25T19:14:05.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.884+0000 7fae99d9b640 1 -- 172.21.15.6:0/784594232 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae88003520 con 0x7fae7c0088b0 2024-03-25T19:14:05.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.916+0000 7fae99d9b640 1 -- 172.21.15.6:0/784594232 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2413356607 0 0) 0x7fae7c00d740 con 0x7fae7c008cf0 2024-03-25T19:14:05.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.916+0000 7fae99d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:14:05.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.916+0000 7fae99d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:05.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.916+0000 7fae99d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:05.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.916+0000 7fae99d9b640 10 monclient(hunting): my global_id is 4102 2024-03-25T19:14:05.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.916+0000 7fae99d9b640 1 -- 172.21.15.6:0/784594232 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae7c00d4f0 con 0x7fae7c008cf0 2024-03-25T19:14:05.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.917+0000 7fae99d9b640 1 -- 172.21.15.6:0/784594232 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3038262843 0 0) 0x7fae7c0081a0 con 0x7fae7c007af0 2024-03-25T19:14:05.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.917+0000 7fae99d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:14:05.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.917+0000 7fae99d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:05.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.917+0000 7fae99d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:05.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.917+0000 7fae99d9b640 10 monclient(hunting): my global_id is 4106 2024-03-25T19:14:05.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.917+0000 7fae99d9b640 1 -- 172.21.15.6:0/784594232 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae7c00d740 con 0x7fae7c007af0 2024-03-25T19:14:05.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.969+0000 7fae99d9b640 1 -- 172.21.15.6:0/784594232 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1080021558 0 0) 0x7fae7c00d4f0 con 0x7fae7c008cf0 2024-03-25T19:14:05.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.969+0000 7fae99d9b640 1 -- 172.21.15.6:0/784594232 >> v1:172.21.15.6:6790/0 conn(0x7fae7c007af0 legacy=0x7fae7c0073e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:05.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.969+0000 7fae99d9b640 1 -- 172.21.15.6:0/784594232 >> v1:172.21.15.6:6789/0 conn(0x7fae7c0088b0 legacy=0x7fae7c00ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:05.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.969+0000 7fae99d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:14:05.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.969+0000 7fae99d9b640 1 monclient: found mon.noname-c 2024-03-25T19:14:05.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.969+0000 7fae99d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:14:05.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.969+0000 7fae99d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:14:05.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.969+0000 7fae99d9b640 1 -- 172.21.15.6:0/784594232 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae7c008380 con 0x7fae7c008cf0 2024-03-25T19:14:05.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.969+0000 7fae99d9b640 10 monclient: _finish_auth 0 2024-03-25T19:14:05.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.969+0000 7fae99d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:14:05.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.969+0000 7fae99d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:14:05.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.969+0000 7fae99d9b640 1 -- 172.21.15.6:0/784594232 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fae84004030 con 0x7fae7c008cf0 2024-03-25T19:14:05.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.969+0000 7faea09cb640 5 monclient: authenticate success, global_id 4102 2024-03-25T19:14:05.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.969+0000 7faea09cb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:14:05.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.969+0000 7fae99d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:05.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.969+0000 7fae99d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:14:05.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.969+0000 7fae99d9b640 10 monclient: dump: 2024-03-25T19:14:05.971 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-03-25T19:14:05.971 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:05.971 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:05.971 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:05.971 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:05.971 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:05.971 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:05.971 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:05.971 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:05.971 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-03-25T19:14:05.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.969+0000 7faea09cb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:14:05.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.998+0000 7fae99d9b640 1 -- 172.21.15.6:0/784594232 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae84004200 con 0x7fae7c008cf0 2024-03-25T19:14:05.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.998+0000 7fae99d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:14:05.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.998+0000 7fae99d9b640 1 -- 172.21.15.6:0/784594232 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fae84004610 con 0x7fae7c008cf0 2024-03-25T19:14:05.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.998+0000 7fae99d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:05.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.998+0000 7fae99d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:14:05.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.998+0000 7fae99d9b640 10 monclient: dump: 2024-03-25T19:14:05.998 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-03-25T19:14:05.998 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:05.998 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:05.998 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:05.998 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:05.999 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:05.999 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:05.999 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:05.999 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:05.999 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-03-25T19:14:05.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.998+0000 7faea09cb640 10 monclient: get_monmap_and_config success 2024-03-25T19:14:05.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.998+0000 7faea09cb640 4 set_mon_vals no callback set 2024-03-25T19:14:06.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.998+0000 7faea09cb640 10 monclient: shutdown 2024-03-25T19:14:06.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.999+0000 7faea09cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:14:06.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.999+0000 7faea09cb640 1 -- 172.21.15.6:0/784594232 >> v1:172.21.15.192:6789/0 conn(0x7fae7c008cf0 legacy=0x7fae7c00f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:06.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.999+0000 7faea09cb640 1 -- 172.21.15.6:0/784594232 shutdown_connections 2024-03-25T19:14:06.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.999+0000 7faea09cb640 1 -- 172.21.15.6:0/784594232 >> 172.21.15.6:0/784594232 conn(0x7fae9c066b00 msgr2=0x7fae9c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:14:06.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.999+0000 7faea09cb640 1 -- 172.21.15.6:0/784594232 shutdown_connections 2024-03-25T19:14:06.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.999+0000 7faea09cb640 1 -- 172.21.15.6:0/784594232 wait complete. 2024-03-25T19:14:06.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.999+0000 7faea09cb640 10 monclient: build_initial_monmap 2024-03-25T19:14:06.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:05.999+0000 7faea09cb640 10 monclient: monmap: 2024-03-25T19:14:06.002 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-03-25T19:14:06.002 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:14:06.002 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-03-25T19:14:06.002 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-03-25T19:14:06.002 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:14:06.002 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-03-25T19:14:06.002 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:14:06.002 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:14:06.003 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:14:06.003 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-03-25T19:14:06.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.000+0000 7faea09cb640 1 librados: starting msgr at 2024-03-25T19:14:06.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.000+0000 7faea09cb640 1 librados: starting objecter 2024-03-25T19:14:06.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.000+0000 7faea09cb640 1 Processor -- start 2024-03-25T19:14:06.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.000+0000 7faea09cb640 1 -- start start 2024-03-25T19:14:06.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.000+0000 7faea09cb640 1 librados: setting wanted keys 2024-03-25T19:14:06.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.000+0000 7faea09cb640 1 librados: calling monclient init 2024-03-25T19:14:06.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.000+0000 7faea09cb640 10 monclient: init 2024-03-25T19:14:06.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.000+0000 7faea09cb640 10 monclient: _reopen_session rank -1 2024-03-25T19:14:06.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.000+0000 7faea09cb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:14:06.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.000+0000 7faea09cb640 10 monclient(hunting): picked mon.noname-b con 0x7fae7c007af0 addr v1:172.21.15.192:6789/0 2024-03-25T19:14:06.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.000+0000 7faea09cb640 10 monclient(hunting): picked mon.noname-a con 0x7fae7c0088b0 addr v1:172.21.15.6:6789/0 2024-03-25T19:14:06.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.000+0000 7faea09cb640 10 monclient(hunting): picked mon.noname-c con 0x7fae7c008cf0 addr v1:172.21.15.6:6790/0 2024-03-25T19:14:06.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.000+0000 7faea09cb640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae9c0acce0 con 0x7fae7c0088b0 2024-03-25T19:14:06.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.000+0000 7faea09cb640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae9c1d82a0 con 0x7fae7c007af0 2024-03-25T19:14:06.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.000+0000 7faea09cb640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae9c1d8430 con 0x7fae7c008cf0 2024-03-25T19:14:06.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.000+0000 7faea09cb640 10 monclient(hunting): _renew_subs 2024-03-25T19:14:06.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.000+0000 7faea09cb640 10 monclient(hunting): authenticate will time out at 5950.240680s 2024-03-25T19:14:06.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.001+0000 7fae9ad9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fae7c008cf0 0x7fae9c1d7b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:47066/0 (socket says 172.21.15.6:47066) 2024-03-25T19:14:06.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.001+0000 7fae9ad9d640 1 -- 172.21.15.6:0/1595133502 learned_addr learned my addr 172.21.15.6:0/1595133502 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:14:06.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.001+0000 7fae92ffd640 10 client.?.objecter ms_handle_connect 0x7fae7c008cf0 2024-03-25T19:14:06.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.001+0000 7fae92ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:06.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.003+0000 7fae92ffd640 10 client.?.objecter ms_handle_connect 0x7fae7c007af0 2024-03-25T19:14:06.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.003+0000 7fae92ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:06.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.024+0000 7fae92ffd640 10 client.?.objecter ms_handle_connect 0x7fae7c0088b0 2024-03-25T19:14:06.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.024+0000 7fae92ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:06.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.056+0000 7fae92ffd640 1 -- 172.21.15.6:0/1595133502 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3874889045 0 0) 0x7fae9c1d82a0 con 0x7fae7c007af0 2024-03-25T19:14:06.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.056+0000 7fae92ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:14:06.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.056+0000 7fae92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:06.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.056+0000 7fae92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:06.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.056+0000 7fae92ffd640 10 monclient(hunting): my global_id is 4111 2024-03-25T19:14:06.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.056+0000 7fae92ffd640 1 -- 172.21.15.6:0/1595133502 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae7c00d4f0 con 0x7fae7c007af0 2024-03-25T19:14:06.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.081+0000 7fae92ffd640 1 -- 172.21.15.6:0/1595133502 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1889380858 0 0) 0x7fae9c1d8430 con 0x7fae7c008cf0 2024-03-25T19:14:06.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.081+0000 7fae92ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:14:06.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.081+0000 7fae92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:06.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.081+0000 7fae92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:06.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.081+0000 7fae92ffd640 10 monclient(hunting): my global_id is 4112 2024-03-25T19:14:06.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.081+0000 7fae92ffd640 1 -- 172.21.15.6:0/1595133502 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae9c1d82a0 con 0x7fae7c008cf0 2024-03-25T19:14:06.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.113+0000 7fae92ffd640 1 -- 172.21.15.6:0/1595133502 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 41467611 0 0) 0x7fae7c00d4f0 con 0x7fae7c007af0 2024-03-25T19:14:06.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.113+0000 7fae92ffd640 1 -- 172.21.15.6:0/1595133502 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae9c1d8430 con 0x7fae7c007af0 2024-03-25T19:14:06.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.113+0000 7fae92ffd640 1 -- 172.21.15.6:0/1595133502 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fae8c0041c0 con 0x7fae7c007af0 2024-03-25T19:14:06.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.113+0000 7fae92ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:06.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.113+0000 7fae92ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:14:06.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.113+0000 7fae92ffd640 10 monclient(hunting): dump: 2024-03-25T19:14:06.113 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-03-25T19:14:06.113 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:06.114 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:06.114 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:06.114 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:06.114 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:06.114 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:06.114 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:06.114 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:06.114 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-03-25T19:14:06.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.113+0000 7fae92ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:14:06.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.118+0000 7fae92ffd640 1 -- 172.21.15.6:0/1595133502 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2754560204 0 0) 0x7fae9c0acce0 con 0x7fae7c0088b0 2024-03-25T19:14:06.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.118+0000 7fae92ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:14:06.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.118+0000 7fae92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:06.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.118+0000 7fae92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:06.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.118+0000 7fae92ffd640 10 monclient(hunting): my global_id is 4113 2024-03-25T19:14:06.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.118+0000 7fae92ffd640 1 -- 172.21.15.6:0/1595133502 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae7c00d4f0 con 0x7fae7c0088b0 2024-03-25T19:14:06.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.139+0000 7fae92ffd640 1 -- 172.21.15.6:0/1595133502 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2147228935 0 0) 0x7fae7c00d4f0 con 0x7fae7c0088b0 2024-03-25T19:14:06.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.139+0000 7fae92ffd640 1 -- 172.21.15.6:0/1595133502 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae9c0acce0 con 0x7fae7c0088b0 2024-03-25T19:14:06.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.139+0000 7fae92ffd640 1 -- 172.21.15.6:0/1595133502 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fae84004510 con 0x7fae7c0088b0 2024-03-25T19:14:06.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.139+0000 7fae92ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:06.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.139+0000 7fae92ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:14:06.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.139+0000 7fae92ffd640 10 monclient(hunting): dump: 2024-03-25T19:14:06.139 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-03-25T19:14:06.139 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:06.139 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:06.139 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:06.139 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:06.139 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:06.140 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:06.140 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:06.140 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:06.140 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-03-25T19:14:06.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.182+0000 7fae92ffd640 1 -- 172.21.15.6:0/1595133502 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3502457569 0 0) 0x7fae9c0acce0 con 0x7fae7c0088b0 2024-03-25T19:14:06.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.182+0000 7fae92ffd640 1 -- 172.21.15.6:0/1595133502 >> v1:172.21.15.6:6790/0 conn(0x7fae7c008cf0 legacy=0x7fae9c1d7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:06.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.182+0000 7fae92ffd640 1 -- 172.21.15.6:0/1595133502 >> v1:172.21.15.192:6789/0 conn(0x7fae7c007af0 legacy=0x7fae9c0ace90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:06.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.182+0000 7fae92ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:14:06.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.182+0000 7fae92ffd640 1 monclient: found mon.a 2024-03-25T19:14:06.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.182+0000 7fae92ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:14:06.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.182+0000 7fae92ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:14:06.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.182+0000 7fae92ffd640 1 -- 172.21.15.6:0/1595133502 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae9c1d85c0 con 0x7fae7c0088b0 2024-03-25T19:14:06.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.182+0000 7fae92ffd640 10 monclient: _finish_auth 0 2024-03-25T19:14:06.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.182+0000 7fae92ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:14:06.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.182+0000 7fae92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:14:06.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.182+0000 7faea09cb640 5 monclient: authenticate success, global_id 4113 2024-03-25T19:14:06.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.182+0000 7faea09cb640 10 monclient: _renew_subs 2024-03-25T19:14:06.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.182+0000 7faea09cb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:14:06.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.182+0000 7faea09cb640 1 -- 172.21.15.6:0/1595133502 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae9c1d8810 con 0x7fae7c0088b0 2024-03-25T19:14:06.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.182+0000 7faea09cb640 10 client.4113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:14:06.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.182+0000 7faea09cb640 10 monclient: _renew_subs 2024-03-25T19:14:06.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.182+0000 7faea09cb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:14:06.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.182+0000 7faea09cb640 1 -- 172.21.15.6:0/1595133502 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fae9c1e36e0 con 0x7fae7c0088b0 2024-03-25T19:14:06.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.182+0000 7faea09cb640 1 librados: init done 2024-03-25T19:14:06.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.183+0000 7faea09cb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:06.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.183+0000 7faea09cb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:06.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.183+0000 7faea09cb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:14:06.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.183+0000 7faea09cb640 1 -- 172.21.15.6:0/1595133502 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae9c140040 con 0x7fae7c0088b0 2024-03-25T19:14:06.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.360+0000 7fae92ffd640 1 -- 172.21.15.6:0/1595133502 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae84004920 con 0x7fae7c0088b0 2024-03-25T19:14:06.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.360+0000 7fae92ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:14:06.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.360+0000 7fae92ffd640 1 -- 172.21.15.6:0/1595133502 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fae84004cb0 con 0x7fae7c0088b0 2024-03-25T19:14:06.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.360+0000 7fae92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:06.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.360+0000 7fae92ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:14:06.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.360+0000 7fae92ffd640 10 monclient: dump: 2024-03-25T19:14:06.361 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-03-25T19:14:06.361 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:06.361 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:06.361 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:06.361 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:06.361 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:06.361 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:06.361 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:06.361 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:06.361 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-03-25T19:14:06.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.360+0000 7fae92ffd640 1 -- 172.21.15.6:0/1595133502 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fae84005260 con 0x7fae7c0088b0 2024-03-25T19:14:06.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.360+0000 7fae9959a640 4 set_mon_vals no callback set 2024-03-25T19:14:06.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.360+0000 7fae92ffd640 1 -- 172.21.15.6:0/1595133502 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 416165132 0 0) 0x7fae84006770 con 0x7fae7c0088b0 2024-03-25T19:14:06.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.360+0000 7fae92ffd640 10 client.4113.objecter ms_dispatch 0x7fae9c000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T19:14:06.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.360+0000 7fae92ffd640 3 client.4113.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T19:14:06.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.360+0000 7fae92ffd640 3 client.4113.objecter handle_osd_map decoding full epoch 1 2024-03-25T19:14:06.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.360+0000 7fae92ffd640 20 client.4113.objecter dump_active .. 0 homeless 2024-03-25T19:14:06.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.360+0000 7fae92ffd640 1 -- 172.21.15.6:0/1595133502 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fae840069e0 con 0x7fae7c0088b0 2024-03-25T19:14:06.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.360+0000 7fae92ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:06.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.360+0000 7fae92ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:14:06.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.434+0000 7faea09cb640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:14:06.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.434+0000 7faea09cb640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:14:06.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.434+0000 7faea09cb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:14:06.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.434+0000 7faea09cb640 1 -- 172.21.15.6:0/1595133502 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fae9c140040 con 0x7fae7c0088b0 2024-03-25T19:14:06.569 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-25T19:14:06.569+0000 7fbad1252040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T19:14:06.630 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-25T19:14:06.630+0000 7fbad1252040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T19:14:06.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.631+0000 7fae92ffd640 1 -- 172.21.15.6:0/1595133502 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7fae84005820 con 0x7fae7c0088b0 2024-03-25T19:14:06.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.631+0000 7fae92ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:14:06.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.631+0000 7fae92ffd640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T19:14:06.631 INFO:teuthology.orchestra.run.smithi006.stderr:adjusted tunables profile to default 2024-03-25T19:14:06.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.632+0000 7faea09cb640 10 librados: watch_flush enter 2024-03-25T19:14:06.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.632+0000 7faea09cb640 10 librados: watch_flush exit 2024-03-25T19:14:06.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.632+0000 7faea09cb640 20 client.4113.objecter shutdown clearing up homeless session... 2024-03-25T19:14:06.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.632+0000 7faea09cb640 10 client.4113.objecter successfully canceled tick 2024-03-25T19:14:06.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.632+0000 7faea09cb640 10 monclient: shutdown 2024-03-25T19:14:06.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.632+0000 7faea09cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:14:06.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.632+0000 7faea09cb640 1 -- 172.21.15.6:0/1595133502 >> v1:172.21.15.6:6789/0 conn(0x7fae7c0088b0 legacy=0x7fae9c0ac5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:06.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.632+0000 7faea09cb640 1 -- 172.21.15.6:0/1595133502 shutdown_connections 2024-03-25T19:14:06.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.632+0000 7faea09cb640 1 -- 172.21.15.6:0/1595133502 >> 172.21.15.6:0/1595133502 conn(0x7fae9c066b00 msgr2=0x7fae9c0b3490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:14:06.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.632+0000 7faea09cb640 1 -- 172.21.15.6:0/1595133502 shutdown_connections 2024-03-25T19:14:06.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.632+0000 7faea09cb640 1 -- 172.21.15.6:0/1595133502 wait complete. 2024-03-25T19:14:06.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:06.632+0000 7faea09cb640 1 librados: shutdown 2024-03-25T19:14:06.643 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T19:14:06.643 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T19:14:06.643 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-25T19:14:06.643 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T19:14:06.671 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-25T19:14:06.671 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T19:14:06.682 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-03-25T19:14:06.682+0000 7f1c54213040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T19:14:06.736 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-03-25T19:14:06.736+0000 7f1c54213040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T19:14:06.739 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-25T19:14:06.739 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T19:14:06.807 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-25T19:14:06.807 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T19:14:06.875 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-25T19:14:06.875 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T19:14:06.903 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-25T19:14:06.903 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T19:14:06.969 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-25T19:14:06.970 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T19:14:07.036 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-25T19:14:07.037 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T19:14:07.103 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new 15732146-8cac-4747-96aa-c2cad384c361 0 2024-03-25T19:14:07.223 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-25T19:14:07.223+0000 7fbad1252040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T19:14:07.243 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.243+0000 7f2a142a1640 10 monclient: get_monmap_and_config 2024-03-25T19:14:07.243 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.243+0000 7f2a142a1640 10 monclient: build_initial_monmap 2024-03-25T19:14:07.243 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.243+0000 7f2a142a1640 10 monclient: monmap: 2024-03-25T19:14:07.243 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2024-03-25T19:14:07.243 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:07.243 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:14:07.244469+0000 2024-03-25T19:14:07.244 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:14:07.244469+0000 2024-03-25T19:14:07.244 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2024-03-25T19:14:07.244 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:07.244 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:14:07.244 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:14:07.244 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:14:07.244 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:07.245 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.244+0000 7f2a142a1640 1 Processor -- start 2024-03-25T19:14:07.245 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.244+0000 7f2a142a1640 1 -- start start 2024-03-25T19:14:07.245 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.244+0000 7f2a142a1640 10 monclient: init 2024-03-25T19:14:07.246 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.244+0000 7f2a142a1640 10 monclient: _reopen_session rank -1 2024-03-25T19:14:07.246 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.244+0000 7f2a142a1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:14:07.246 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.244+0000 7f2a142a1640 10 monclient(hunting): picked mon.noname-a con 0x7f2a0c0b4b50 addr v1:172.21.15.6:6789/0 2024-03-25T19:14:07.246 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.244+0000 7f2a142a1640 10 monclient(hunting): picked mon.noname-b con 0x7f2a0c13ff20 addr v1:172.21.15.6:6790/0 2024-03-25T19:14:07.246 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.244+0000 7f2a142a1640 10 monclient(hunting): picked mon.noname-c con 0x7f2a0c142dd0 addr v1:172.21.15.192:6789/0 2024-03-25T19:14:07.246 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.244+0000 7f2a142a1640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a0c0ab440 con 0x7f2a0c0b4b50 2024-03-25T19:14:07.246 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.244+0000 7f2a142a1640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a0c0ab6b0 con 0x7f2a0c142dd0 2024-03-25T19:14:07.246 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.244+0000 7f2a142a1640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a0c0a9f10 con 0x7f2a0c13ff20 2024-03-25T19:14:07.246 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.244+0000 7f2a142a1640 10 monclient(hunting): _renew_subs 2024-03-25T19:14:07.246 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.244+0000 7f2a142a1640 10 monclient(hunting): authenticate will time out at 5969.538285s 2024-03-25T19:14:07.246 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.245+0000 7f2a1203d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2a0c0b4b50 0x7f2a0c13f810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.192:42774/0 (socket says 172.21.15.192:42774) 2024-03-25T19:14:07.247 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.245+0000 7f2a1203d640 1 -- 172.21.15.192:0/2734030210 learned_addr learned my addr 172.21.15.192:0/2734030210 (peer_addr_for_me v1:172.21.15.192:0/0) 2024-03-25T19:14:07.248 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.247+0000 7f2a1103b640 1 -- 172.21.15.192:0/2734030210 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1178142880 0 0) 0x7f2a0c0a9f10 con 0x7f2a0c13ff20 2024-03-25T19:14:07.248 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.248+0000 7f2a1103b640 10 monclient(hunting): _negotiate 2024-03-25T19:14:07.248 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.248+0000 7f2a1103b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:07.249 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.248+0000 7f2a1103b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:07.249 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.248+0000 7f2a1103b640 10 monclient(hunting): my global_id is 4115 2024-03-25T19:14:07.249 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.248+0000 7f2a1103b640 1 -- 172.21.15.192:0/2734030210 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29f8003520 con 0x7f2a0c13ff20 2024-03-25T19:14:07.249 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.248+0000 7f2a1103b640 1 -- 172.21.15.192:0/2734030210 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2235917522 0 0) 0x7f2a0c0ab440 con 0x7f2a0c0b4b50 2024-03-25T19:14:07.249 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.248+0000 7f2a1103b640 10 monclient(hunting): _negotiate 2024-03-25T19:14:07.249 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.248+0000 7f2a1103b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:07.249 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.248+0000 7f2a1103b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:07.249 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.248+0000 7f2a1103b640 10 monclient(hunting): my global_id is 4116 2024-03-25T19:14:07.249 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.248+0000 7f2a1103b640 1 -- 172.21.15.192:0/2734030210 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a0c0a9f10 con 0x7f2a0c0b4b50 2024-03-25T19:14:07.249 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.248+0000 7f2a1103b640 1 -- 172.21.15.192:0/2734030210 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1004794621 0 0) 0x7f2a0c0ab6b0 con 0x7f2a0c142dd0 2024-03-25T19:14:07.249 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.248+0000 7f2a1103b640 10 monclient(hunting): _negotiate 2024-03-25T19:14:07.249 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.248+0000 7f2a1103b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:07.249 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.248+0000 7f2a1103b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:07.249 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.248+0000 7f2a1103b640 10 monclient(hunting): my global_id is 4114 2024-03-25T19:14:07.249 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.248+0000 7f2a1103b640 1 -- 172.21.15.192:0/2734030210 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a0c0ab440 con 0x7f2a0c142dd0 2024-03-25T19:14:07.251 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.250+0000 7f2a1103b640 1 -- 172.21.15.192:0/2734030210 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2469637139 0 0) 0x7f2a0c0a9f10 con 0x7f2a0c0b4b50 2024-03-25T19:14:07.251 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.250+0000 7f2a1103b640 1 -- 172.21.15.192:0/2734030210 >> v1:172.21.15.6:6790/0 conn(0x7f2a0c13ff20 legacy=0x7f2a0c1426c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:07.251 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.250+0000 7f2a1103b640 1 -- 172.21.15.192:0/2734030210 >> v1:172.21.15.192:6789/0 conn(0x7f2a0c142dd0 legacy=0x7f2a0c14df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:07.251 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.250+0000 7f2a1103b640 10 monclient: _finish_hunting 0 2024-03-25T19:14:07.251 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.250+0000 7f2a1103b640 1 monclient: found mon.noname-a 2024-03-25T19:14:07.251 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.250+0000 7f2a1103b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:14:07.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.250+0000 7f2a1103b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-03-25T19:14:07.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.250+0000 7f2a1103b640 1 -- 172.21.15.192:0/2734030210 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a0c0aa0a0 con 0x7f2a0c0b4b50 2024-03-25T19:14:07.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.250+0000 7f2a1103b640 10 monclient: _finish_auth 0 2024-03-25T19:14:07.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.250+0000 7f2a1103b640 10 monclient: _check_auth_tickets 2024-03-25T19:14:07.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.250+0000 7f2a1103b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:14:07.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.250+0000 7f2a1103b640 1 -- 172.21.15.192:0/2734030210 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f2a08001670 con 0x7f2a0c0b4b50 2024-03-25T19:14:07.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.250+0000 7f2a1103b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:07.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.250+0000 7f2a1103b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:14:07.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.250+0000 7f2a1103b640 10 monclient: dump: 2024-03-25T19:14:07.252 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:07.252 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:07.252 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:07.252 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:07.252 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:07.252 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:07.252 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:07.253 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:07.253 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:07.253 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:07.253 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.250+0000 7f2a142a1640 5 monclient: authenticate success, global_id 4116 2024-03-25T19:14:07.253 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.250+0000 7f2a142a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:14:07.253 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.253+0000 7f2a1103b640 1 -- 172.21.15.192:0/2734030210 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a080017d0 con 0x7f2a0c0b4b50 2024-03-25T19:14:07.253 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.253+0000 7f2a1103b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:14:07.254 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.253+0000 7f2a1103b640 1 -- 172.21.15.192:0/2734030210 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f2a08001b60 con 0x7f2a0c0b4b50 2024-03-25T19:14:07.254 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.253+0000 7f2a1103b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:07.254 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.253+0000 7f2a1103b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:14:07.254 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.253+0000 7f2a1103b640 10 monclient: dump: 2024-03-25T19:14:07.254 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:07.254 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:07.254 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:07.254 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:07.254 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:07.254 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:07.254 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:07.254 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:07.254 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:07.255 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:07.255 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.253+0000 7f2a142a1640 10 monclient: get_monmap_and_config success 2024-03-25T19:14:07.255 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.253+0000 7f2a142a1640 4 set_mon_vals no callback set 2024-03-25T19:14:07.257 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.254+0000 7f2a142a1640 10 monclient: shutdown 2024-03-25T19:14:07.257 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.254+0000 7f2a142a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:14:07.257 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.254+0000 7f2a142a1640 1 -- 172.21.15.192:0/2734030210 >> v1:172.21.15.6:6789/0 conn(0x7f2a0c0b4b50 legacy=0x7f2a0c13f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:07.257 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.254+0000 7f2a142a1640 1 -- 172.21.15.192:0/2734030210 shutdown_connections 2024-03-25T19:14:07.257 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.254+0000 7f2a142a1640 1 -- 172.21.15.192:0/2734030210 >> 172.21.15.192:0/2734030210 conn(0x7f2a0c066920 msgr2=0x7f2a0c066d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:14:07.258 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.254+0000 7f2a142a1640 1 -- 172.21.15.192:0/2734030210 shutdown_connections 2024-03-25T19:14:07.258 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.254+0000 7f2a142a1640 1 -- 172.21.15.192:0/2734030210 wait complete. 2024-03-25T19:14:07.258 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.255+0000 7f2a142a1640 10 monclient: build_initial_monmap 2024-03-25T19:14:07.258 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.255+0000 7f2a142a1640 10 monclient: monmap: 2024-03-25T19:14:07.258 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2024-03-25T19:14:07.258 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:14:07.258 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 0.000000 2024-03-25T19:14:07.258 INFO:teuthology.orchestra.run.smithi192.stderr:created 0.000000 2024-03-25T19:14:07.258 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2024-03-25T19:14:07.258 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2024-03-25T19:14:07.258 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:14:07.258 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:14:07.258 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:14:07.258 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:07.258 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.255+0000 7f2a142a1640 1 librados: starting msgr at 2024-03-25T19:14:07.259 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.255+0000 7f2a142a1640 1 librados: starting objecter 2024-03-25T19:14:07.259 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.255+0000 7f2a142a1640 1 Processor -- start 2024-03-25T19:14:07.259 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.255+0000 7f2a142a1640 1 -- start start 2024-03-25T19:14:07.259 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.255+0000 7f2a142a1640 1 librados: setting wanted keys 2024-03-25T19:14:07.259 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.255+0000 7f2a142a1640 1 librados: calling monclient init 2024-03-25T19:14:07.259 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.255+0000 7f2a142a1640 10 monclient: init 2024-03-25T19:14:07.259 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.256+0000 7f2a142a1640 10 monclient: _reopen_session rank -1 2024-03-25T19:14:07.259 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.256+0000 7f2a142a1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:14:07.259 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.256+0000 7f2a142a1640 10 monclient(hunting): picked mon.noname-b con 0x7f2a0c0b4b50 addr v1:172.21.15.192:6789/0 2024-03-25T19:14:07.259 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.256+0000 7f2a142a1640 10 monclient(hunting): picked mon.noname-a con 0x7f2a0c13ff20 addr v1:172.21.15.6:6789/0 2024-03-25T19:14:07.259 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.256+0000 7f2a142a1640 10 monclient(hunting): picked mon.noname-c con 0x7f2a0c142dd0 addr v1:172.21.15.6:6790/0 2024-03-25T19:14:07.259 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.256+0000 7f2a142a1640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a0c144510 con 0x7f2a0c13ff20 2024-03-25T19:14:07.259 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.256+0000 7f2a142a1640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a0c140430 con 0x7f2a0c0b4b50 2024-03-25T19:14:07.259 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.256+0000 7f2a142a1640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a0c1405c0 con 0x7f2a0c142dd0 2024-03-25T19:14:07.260 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.256+0000 7f2a142a1640 10 monclient(hunting): _renew_subs 2024-03-25T19:14:07.260 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.256+0000 7f2a142a1640 10 monclient(hunting): authenticate will time out at 5969.549566s 2024-03-25T19:14:07.260 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.256+0000 7f2a1183c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2a0c13ff20 0x7f2a0c145ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.192:42790/0 (socket says 172.21.15.192:42790) 2024-03-25T19:14:07.260 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.256+0000 7f2a1183c640 1 -- 172.21.15.192:0/920770340 learned_addr learned my addr 172.21.15.192:0/920770340 (peer_addr_for_me v1:172.21.15.192:0/0) 2024-03-25T19:14:07.260 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.256+0000 7f2a02ffd640 10 client.?.objecter ms_handle_connect 0x7f2a0c13ff20 2024-03-25T19:14:07.260 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.256+0000 7f2a02ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:07.260 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.256+0000 7f2a02ffd640 10 client.?.objecter ms_handle_connect 0x7f2a0c0b4b50 2024-03-25T19:14:07.260 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.256+0000 7f2a02ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:07.261 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.258+0000 7f2a02ffd640 1 -- 172.21.15.192:0/920770340 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4233512007 0 0) 0x7f2a0c140430 con 0x7f2a0c0b4b50 2024-03-25T19:14:07.261 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.258+0000 7f2a02ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:14:07.261 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.258+0000 7f2a02ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:07.261 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.258+0000 7f2a02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:07.261 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.258+0000 7f2a02ffd640 10 monclient(hunting): my global_id is 4117 2024-03-25T19:14:07.261 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.258+0000 7f2a02ffd640 1 -- 172.21.15.192:0/920770340 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29f0004eb0 con 0x7f2a0c0b4b50 2024-03-25T19:14:07.261 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.259+0000 7f2a02ffd640 1 -- 172.21.15.192:0/920770340 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1767061615 0 0) 0x7f2a0c144510 con 0x7f2a0c13ff20 2024-03-25T19:14:07.261 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.259+0000 7f2a02ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:14:07.261 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.259+0000 7f2a02ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:07.262 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.259+0000 7f2a02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:07.262 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.259+0000 7f2a02ffd640 10 monclient(hunting): my global_id is 4119 2024-03-25T19:14:07.262 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.259+0000 7f2a02ffd640 1 -- 172.21.15.192:0/920770340 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a0c140430 con 0x7f2a0c13ff20 2024-03-25T19:14:07.263 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.263+0000 7f2a02ffd640 1 -- 172.21.15.192:0/920770340 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2907851418 0 0) 0x7f29f0004eb0 con 0x7f2a0c0b4b50 2024-03-25T19:14:07.263 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.263+0000 7f2a02ffd640 1 -- 172.21.15.192:0/920770340 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a0c144510 con 0x7f2a0c0b4b50 2024-03-25T19:14:07.263 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.263+0000 7f2a02ffd640 1 -- 172.21.15.192:0/920770340 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f2a08001ae0 con 0x7f2a0c0b4b50 2024-03-25T19:14:07.263 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.263+0000 7f2a02ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:07.263 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.263+0000 7f2a02ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:14:07.263 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.263+0000 7f2a02ffd640 10 monclient(hunting): dump: 2024-03-25T19:14:07.263 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:07.264 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:07.264 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:07.264 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:07.264 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:07.264 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:07.264 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:07.264 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:07.264 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:07.264 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:07.264 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.263+0000 7f2a02ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:14:07.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.264+0000 7f2a02ffd640 1 -- 172.21.15.192:0/920770340 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 658296887 0 0) 0x7f2a0c140430 con 0x7f2a0c13ff20 2024-03-25T19:14:07.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.264+0000 7f2a02ffd640 1 -- 172.21.15.192:0/920770340 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29f0004eb0 con 0x7f2a0c13ff20 2024-03-25T19:14:07.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.264+0000 7f2a02ffd640 1 -- 172.21.15.192:0/920770340 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f29fc001990 con 0x7f2a0c13ff20 2024-03-25T19:14:07.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.264+0000 7f2a02ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:07.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.264+0000 7f2a02ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:14:07.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.264+0000 7f2a02ffd640 10 monclient(hunting): dump: 2024-03-25T19:14:07.265 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:07.265 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:07.265 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:07.265 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:07.265 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:07.265 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:07.265 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:07.266 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:07.266 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:07.266 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:07.267 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.266+0000 7f2a02ffd640 1 -- 172.21.15.192:0/920770340 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1581753593 0 0) 0x7f2a0c144510 con 0x7f2a0c0b4b50 2024-03-25T19:14:07.267 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.266+0000 7f2a02ffd640 1 -- 172.21.15.192:0/920770340 >> v1:172.21.15.6:6790/0 conn(0x7f2a0c142dd0 legacy=0x7f2a0c1d7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:07.267 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.266+0000 7f2a02ffd640 1 -- 172.21.15.192:0/920770340 >> v1:172.21.15.6:6789/0 conn(0x7f2a0c13ff20 legacy=0x7f2a0c145ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:07.267 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.266+0000 7f2a02ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:14:07.267 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.266+0000 7f2a02ffd640 1 monclient: found mon.b 2024-03-25T19:14:07.268 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.266+0000 7f2a02ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:14:07.268 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.266+0000 7f2a02ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:07.268 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.266+0000 7f2a02ffd640 1 -- 172.21.15.192:0/920770340 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a0c134c30 con 0x7f2a0c0b4b50 2024-03-25T19:14:07.268 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.266+0000 7f2a02ffd640 10 monclient: _finish_auth 0 2024-03-25T19:14:07.268 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.266+0000 7f2a02ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:14:07.268 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.266+0000 7f2a02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:14:07.268 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.266+0000 7f2a142a1640 5 monclient: authenticate success, global_id 4117 2024-03-25T19:14:07.268 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.266+0000 7f2a142a1640 10 monclient: _renew_subs 2024-03-25T19:14:07.268 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.266+0000 7f2a142a1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:07.268 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.266+0000 7f2a142a1640 1 -- 172.21.15.192:0/920770340 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a0c141e40 con 0x7f2a0c0b4b50 2024-03-25T19:14:07.268 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.267+0000 7f2a142a1640 10 client.4117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:14:07.268 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.267+0000 7f2a142a1640 10 monclient: _renew_subs 2024-03-25T19:14:07.268 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.267+0000 7f2a142a1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:07.268 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.267+0000 7f2a142a1640 1 -- 172.21.15.192:0/920770340 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a0c1467d0 con 0x7f2a0c0b4b50 2024-03-25T19:14:07.268 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.267+0000 7f2a142a1640 1 librados: init done 2024-03-25T19:14:07.269 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.268+0000 7f2a142a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:07.269 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.268+0000 7f2a142a1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:07.269 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.268+0000 7f2a142a1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:07.269 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.268+0000 7f2a142a1640 1 -- 172.21.15.192:0/920770340 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a0c1423e0 con 0x7f2a0c0b4b50 2024-03-25T19:14:07.284 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-25T19:14:07.285+0000 7fbad1252040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T19:14:07.297 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-03-25T19:14:07.297+0000 7f1c54213040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T19:14:07.333 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-25T19:14:07.333+0000 7fbad1252040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T19:14:07.353 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-03-25T19:14:07.353+0000 7f1c54213040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T19:14:07.389 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-25T19:14:07.389+0000 7fbad1252040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T19:14:07.399 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-03-25T19:14:07.400+0000 7f1c54213040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T19:14:07.433 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-25T19:14:07.433+0000 7fbad1252040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T19:14:07.455 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-03-25T19:14:07.456+0000 7f1c54213040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T19:14:07.464 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.463+0000 7f2a02ffd640 1 -- 172.21.15.192:0/920770340 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a08004a90 con 0x7f2a0c0b4b50 2024-03-25T19:14:07.464 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.463+0000 7f2a02ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:14:07.464 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.463+0000 7f2a02ffd640 1 -- 172.21.15.192:0/920770340 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f2a08004da0 con 0x7f2a0c0b4b50 2024-03-25T19:14:07.464 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.463+0000 7f2a02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:07.464 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.463+0000 7f2a02ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:14:07.464 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.463+0000 7f2a02ffd640 10 monclient: dump: 2024-03-25T19:14:07.464 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:07.465 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:07.465 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:07.465 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:07.465 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:07.465 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:07.465 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:07.465 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:07.465 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:07.465 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:07.465 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.463+0000 7f2a02ffd640 1 -- 172.21.15.192:0/920770340 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2a08005290 con 0x7f2a0c0b4b50 2024-03-25T19:14:07.465 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.463+0000 7f2a02ffd640 1 -- 172.21.15.192:0/920770340 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 2037017242 0 0) 0x7f2a080066d0 con 0x7f2a0c0b4b50 2024-03-25T19:14:07.465 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.463+0000 7f2a02ffd640 10 client.4117.objecter ms_dispatch 0x7f2a0c000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T19:14:07.465 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.463+0000 7f2a02ffd640 3 client.4117.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T19:14:07.465 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.463+0000 7f2a02ffd640 3 client.4117.objecter handle_osd_map decoding full epoch 2 2024-03-25T19:14:07.465 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.463+0000 7f2a1083a640 4 set_mon_vals no callback set 2024-03-25T19:14:07.465 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.464+0000 7f2a02ffd640 20 client.4117.objecter dump_active .. 0 homeless 2024-03-25T19:14:07.466 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.464+0000 7f2a02ffd640 1 -- 172.21.15.192:0/920770340 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2a08006ab0 con 0x7f2a0c0b4b50 2024-03-25T19:14:07.466 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.464+0000 7f2a02ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:07.466 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.464+0000 7f2a02ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:14:07.499 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-03-25T19:14:07.499+0000 7f1c54213040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T19:14:07.535 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.535+0000 7f2a142a1640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "15732146-8cac-4747-96aa-c2cad384c361", "id": 0}] 2024-03-25T19:14:07.535 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.535+0000 7f2a142a1640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "15732146-8cac-4747-96aa-c2cad384c361", "id": 0}] 2024-03-25T19:14:07.535 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.535+0000 7f2a142a1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:07.535 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.535+0000 7f2a142a1640 1 -- 172.21.15.192:0/920770340 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd new", "uuid": "15732146-8cac-4747-96aa-c2cad384c361", "id": 0} v 0) v1 -- 0x7f2a0c04f730 con 0x7f2a0c0b4b50 2024-03-25T19:14:07.553 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-25T19:14:07.554+0000 7fbad1252040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T19:14:07.617 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-03-25T19:14:07.616+0000 7f1c54213040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T19:14:07.627 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-25T19:14:07.627+0000 7fbad1252040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T19:14:07.680 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-25T19:14:07.680+0000 7fbad1252040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T19:14:07.688 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-03-25T19:14:07.688+0000 7f1c54213040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T19:14:07.724 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.724+0000 7f2a02ffd640 1 -- 172.21.15.192:0/920770340 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "15732146-8cac-4747-96aa-c2cad384c361", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 884806232 0 732707515) 0x7f2a080057a0 con 0x7f2a0c0b4b50 2024-03-25T19:14:07.724 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.724+0000 7f2a02ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "15732146-8cac-4747-96aa-c2cad384c361", "id": 0}] 2024-03-25T19:14:07.724 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.724+0000 7f2a02ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:14:07.724 INFO:teuthology.orchestra.run.smithi192.stdout:0 2024-03-25T19:14:07.725 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.725+0000 7f2a142a1640 10 librados: watch_flush enter 2024-03-25T19:14:07.725 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.725+0000 7f2a142a1640 10 librados: watch_flush exit 2024-03-25T19:14:07.725 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.725+0000 7f2a142a1640 20 client.4117.objecter shutdown clearing up homeless session... 2024-03-25T19:14:07.726 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.725+0000 7f2a142a1640 10 client.4117.objecter successfully canceled tick 2024-03-25T19:14:07.726 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.725+0000 7f2a142a1640 10 monclient: shutdown 2024-03-25T19:14:07.726 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.725+0000 7f2a142a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:14:07.726 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.725+0000 7f2a142a1640 1 -- 172.21.15.192:0/920770340 >> v1:172.21.15.192:6789/0 conn(0x7f2a0c0b4b50 legacy=0x7f2a0c141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:07.726 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.725+0000 7f2a142a1640 1 -- 172.21.15.192:0/920770340 shutdown_connections 2024-03-25T19:14:07.726 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.725+0000 7f2a142a1640 1 -- 172.21.15.192:0/920770340 >> 172.21.15.192:0/920770340 conn(0x7f2a0c066920 msgr2=0x7f2a0c0a9660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:14:07.726 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.725+0000 7f2a142a1640 1 -- 172.21.15.192:0/920770340 shutdown_connections 2024-03-25T19:14:07.726 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.725+0000 7f2a142a1640 1 -- 172.21.15.192:0/920770340 wait complete. 2024-03-25T19:14:07.726 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.725+0000 7f2a142a1640 1 librados: shutdown 2024-03-25T19:14:07.735 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new 2a83e0b2-dbc3-40be-8026-48e14034e957 1 2024-03-25T19:14:07.737 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-25T19:14:07.737+0000 7fbad1252040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T19:14:07.738 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-03-25T19:14:07.737+0000 7f1c54213040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T19:14:07.786 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-25T19:14:07.787+0000 7fbad1252040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T19:14:07.794 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-03-25T19:14:07.794+0000 7f1c54213040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T19:14:07.837 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.837+0000 7f2059e4e640 10 monclient: get_monmap_and_config 2024-03-25T19:14:07.837 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.837+0000 7f2059e4e640 10 monclient: build_initial_monmap 2024-03-25T19:14:07.837 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.837+0000 7f2059e4e640 10 monclient: monmap: 2024-03-25T19:14:07.837 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2024-03-25T19:14:07.837 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:07.837 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:14:07.838090+0000 2024-03-25T19:14:07.837 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:14:07.838090+0000 2024-03-25T19:14:07.837 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2024-03-25T19:14:07.837 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:07.837 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:14:07.837 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:14:07.838 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:14:07.838 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:07.839 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.837+0000 7f2059e4e640 1 Processor -- start 2024-03-25T19:14:07.839 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.837+0000 7f2059e4e640 1 -- start start 2024-03-25T19:14:07.839 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.837+0000 7f2059e4e640 10 monclient: init 2024-03-25T19:14:07.839 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.838+0000 7f2059e4e640 10 monclient: _reopen_session rank -1 2024-03-25T19:14:07.839 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.838+0000 7f2059e4e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:14:07.839 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.838+0000 7f2059e4e640 10 monclient(hunting): picked mon.noname-a con 0x7f20540b4b50 addr v1:172.21.15.6:6789/0 2024-03-25T19:14:07.839 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.838+0000 7f2059e4e640 10 monclient(hunting): picked mon.noname-b con 0x7f205413ff20 addr v1:172.21.15.6:6790/0 2024-03-25T19:14:07.839 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.838+0000 7f2059e4e640 10 monclient(hunting): picked mon.noname-c con 0x7f2054142dd0 addr v1:172.21.15.192:6789/0 2024-03-25T19:14:07.840 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.838+0000 7f2059e4e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20540ab440 con 0x7f20540b4b50 2024-03-25T19:14:07.840 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.838+0000 7f2059e4e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20540ab6b0 con 0x7f2054142dd0 2024-03-25T19:14:07.840 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.838+0000 7f2059e4e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20540a9f10 con 0x7f205413ff20 2024-03-25T19:14:07.840 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.838+0000 7f2059e4e640 10 monclient(hunting): _renew_subs 2024-03-25T19:14:07.840 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.838+0000 7f2059e4e640 10 monclient(hunting): authenticate will time out at 5970.131960s 2024-03-25T19:14:07.840 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.839+0000 7f20537fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f20540b4b50 0x7f205413f810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.192:42792/0 (socket says 172.21.15.192:42792) 2024-03-25T19:14:07.840 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.839+0000 7f20537fe640 1 -- 172.21.15.192:0/734853857 learned_addr learned my addr 172.21.15.192:0/734853857 (peer_addr_for_me v1:172.21.15.192:0/0) 2024-03-25T19:14:07.841 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.841+0000 7f20527fc640 1 -- 172.21.15.192:0/734853857 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2773588874 0 0) 0x7f20540a9f10 con 0x7f205413ff20 2024-03-25T19:14:07.842 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.841+0000 7f20527fc640 10 monclient(hunting): _negotiate 2024-03-25T19:14:07.842 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.841+0000 7f20527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:07.842 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.841+0000 7f20527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:07.842 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.841+0000 7f20527fc640 10 monclient(hunting): my global_id is 4118 2024-03-25T19:14:07.842 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.841+0000 7f20527fc640 1 -- 172.21.15.192:0/734853857 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2040003520 con 0x7f205413ff20 2024-03-25T19:14:07.842 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.841+0000 7f20527fc640 1 -- 172.21.15.192:0/734853857 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3567275627 0 0) 0x7f20540ab6b0 con 0x7f2054142dd0 2024-03-25T19:14:07.842 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.841+0000 7f20527fc640 10 monclient(hunting): _negotiate 2024-03-25T19:14:07.842 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.841+0000 7f20527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:07.842 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.841+0000 7f20527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:07.842 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.841+0000 7f20527fc640 10 monclient(hunting): my global_id is 4120 2024-03-25T19:14:07.842 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.841+0000 7f20527fc640 1 -- 172.21.15.192:0/734853857 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20540a9f10 con 0x7f2054142dd0 2024-03-25T19:14:07.842 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.842+0000 7f20527fc640 1 -- 172.21.15.192:0/734853857 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 881605410 0 0) 0x7f20540ab440 con 0x7f20540b4b50 2024-03-25T19:14:07.843 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.842+0000 7f20527fc640 10 monclient(hunting): _negotiate 2024-03-25T19:14:07.843 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.842+0000 7f20527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:07.843 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.842+0000 7f20527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:07.843 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.842+0000 7f20527fc640 10 monclient(hunting): my global_id is 4122 2024-03-25T19:14:07.843 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.842+0000 7f20527fc640 1 -- 172.21.15.192:0/734853857 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20540ab6b0 con 0x7f20540b4b50 2024-03-25T19:14:07.845 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.844+0000 7f20527fc640 1 -- 172.21.15.192:0/734853857 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 799098999 0 0) 0x7f20540a9f10 con 0x7f2054142dd0 2024-03-25T19:14:07.845 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.844+0000 7f20527fc640 1 -- 172.21.15.192:0/734853857 >> v1:172.21.15.6:6790/0 conn(0x7f205413ff20 legacy=0x7f20541426c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:07.845 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.844+0000 7f20527fc640 1 -- 172.21.15.192:0/734853857 >> v1:172.21.15.6:6789/0 conn(0x7f20540b4b50 legacy=0x7f205413f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:07.845 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.844+0000 7f20527fc640 10 monclient: _finish_hunting 0 2024-03-25T19:14:07.845 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.844+0000 7f20527fc640 1 monclient: found mon.noname-c 2024-03-25T19:14:07.845 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.844+0000 7f20527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:14:07.845 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.844+0000 7f20527fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:14:07.845 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.844+0000 7f20527fc640 1 -- 172.21.15.192:0/734853857 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20540aa0a0 con 0x7f2054142dd0 2024-03-25T19:14:07.845 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.844+0000 7f20527fc640 10 monclient: _finish_auth 0 2024-03-25T19:14:07.845 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.844+0000 7f20527fc640 10 monclient: _check_auth_tickets 2024-03-25T19:14:07.845 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.844+0000 7f20527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:14:07.845 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.844+0000 7f2059e4e640 5 monclient: authenticate success, global_id 4120 2024-03-25T19:14:07.846 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.844+0000 7f2059e4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:14:07.846 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-03-25T19:14:07.844+0000 7f1c54213040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T19:14:07.847 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.845+0000 7f20527fc640 1 -- 172.21.15.192:0/734853857 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f2044001630 con 0x7f2054142dd0 2024-03-25T19:14:07.847 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.845+0000 7f20527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:07.847 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.845+0000 7f20527fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:14:07.848 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.845+0000 7f20527fc640 10 monclient: dump: 2024-03-25T19:14:07.848 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:07.848 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:07.848 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:07.848 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:07.848 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:07.848 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:07.848 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:07.848 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:07.848 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:07.848 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:07.848 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.845+0000 7f2059e4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:14:07.848 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.846+0000 7f20527fc640 1 -- 172.21.15.192:0/734853857 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2044001810 con 0x7f2054142dd0 2024-03-25T19:14:07.848 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.846+0000 7f20527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:14:07.849 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.846+0000 7f20527fc640 1 -- 172.21.15.192:0/734853857 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f2044001ba0 con 0x7f2054142dd0 2024-03-25T19:14:07.849 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.846+0000 7f2059e4e640 10 monclient: get_monmap_and_config success 2024-03-25T19:14:07.849 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.846+0000 7f2059e4e640 4 set_mon_vals no callback set 2024-03-25T19:14:07.849 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.846+0000 7f20527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:07.849 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.846+0000 7f20527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:14:07.849 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.846+0000 7f20527fc640 10 monclient: dump: 2024-03-25T19:14:07.849 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:07.849 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:07.849 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:07.849 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:07.849 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:07.850 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:07.850 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:07.850 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:07.850 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:07.850 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:07.850 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.847+0000 7f2059e4e640 10 monclient: shutdown 2024-03-25T19:14:07.850 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.847+0000 7f2059e4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:14:07.850 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.847+0000 7f2059e4e640 1 -- 172.21.15.192:0/734853857 >> v1:172.21.15.192:6789/0 conn(0x7f2054142dd0 legacy=0x7f205414df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:07.850 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.847+0000 7f2059e4e640 1 -- 172.21.15.192:0/734853857 shutdown_connections 2024-03-25T19:14:07.850 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.847+0000 7f2059e4e640 1 -- 172.21.15.192:0/734853857 >> 172.21.15.192:0/734853857 conn(0x7f2054066920 msgr2=0x7f2054066d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:14:07.850 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.847+0000 7f2059e4e640 1 -- 172.21.15.192:0/734853857 shutdown_connections 2024-03-25T19:14:07.850 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.847+0000 7f2059e4e640 1 -- 172.21.15.192:0/734853857 wait complete. 2024-03-25T19:14:07.853 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.848+0000 7f2059e4e640 10 monclient: build_initial_monmap 2024-03-25T19:14:07.853 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.848+0000 7f2059e4e640 10 monclient: monmap: 2024-03-25T19:14:07.853 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2024-03-25T19:14:07.853 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:14:07.853 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 0.000000 2024-03-25T19:14:07.853 INFO:teuthology.orchestra.run.smithi192.stderr:created 0.000000 2024-03-25T19:14:07.853 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2024-03-25T19:14:07.854 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2024-03-25T19:14:07.854 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:14:07.854 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:14:07.854 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:14:07.854 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:07.854 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.848+0000 7f2059e4e640 1 librados: starting msgr at 2024-03-25T19:14:07.854 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.848+0000 7f2059e4e640 1 librados: starting objecter 2024-03-25T19:14:07.854 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.848+0000 7f2059e4e640 1 Processor -- start 2024-03-25T19:14:07.854 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.848+0000 7f2059e4e640 1 -- start start 2024-03-25T19:14:07.854 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.848+0000 7f2059e4e640 1 librados: setting wanted keys 2024-03-25T19:14:07.854 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.848+0000 7f2059e4e640 1 librados: calling monclient init 2024-03-25T19:14:07.854 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.848+0000 7f2059e4e640 10 monclient: init 2024-03-25T19:14:07.854 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.849+0000 7f2059e4e640 10 monclient: _reopen_session rank -1 2024-03-25T19:14:07.854 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.849+0000 7f2059e4e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:14:07.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.849+0000 7f2059e4e640 10 monclient(hunting): picked mon.noname-c con 0x7f20540b4b50 addr v1:172.21.15.6:6790/0 2024-03-25T19:14:07.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.849+0000 7f2059e4e640 10 monclient(hunting): picked mon.noname-a con 0x7f205413ff20 addr v1:172.21.15.6:6789/0 2024-03-25T19:14:07.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.849+0000 7f2059e4e640 10 monclient(hunting): picked mon.noname-b con 0x7f2054142dd0 addr v1:172.21.15.192:6789/0 2024-03-25T19:14:07.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.849+0000 7f2059e4e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20541439b0 con 0x7f205413ff20 2024-03-25T19:14:07.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.849+0000 7f2059e4e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2054143b40 con 0x7f2054142dd0 2024-03-25T19:14:07.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.849+0000 7f2059e4e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f205413eae0 con 0x7f20540b4b50 2024-03-25T19:14:07.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.849+0000 7f2059e4e640 10 monclient(hunting): _renew_subs 2024-03-25T19:14:07.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.849+0000 7f2059e4e640 10 monclient(hunting): authenticate will time out at 5970.142730s 2024-03-25T19:14:07.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.849+0000 7f2052ffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f205413ff20 0x7f2054139b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.192:42796/0 (socket says 172.21.15.192:42796) 2024-03-25T19:14:07.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.849+0000 7f2052ffd640 1 -- 172.21.15.192:0/934429503 learned_addr learned my addr 172.21.15.192:0/934429503 (peer_addr_for_me v1:172.21.15.192:0/0) 2024-03-25T19:14:07.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.850+0000 7f204bfff640 10 client.?.objecter ms_handle_connect 0x7f205413ff20 2024-03-25T19:14:07.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.850+0000 7f204bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:07.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.850+0000 7f204bfff640 10 client.?.objecter ms_handle_connect 0x7f20540b4b50 2024-03-25T19:14:07.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.850+0000 7f204bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:07.856 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.850+0000 7f204bfff640 10 client.?.objecter ms_handle_connect 0x7f2054142dd0 2024-03-25T19:14:07.856 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.850+0000 7f204bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:07.856 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.852+0000 7f204bfff640 1 -- 172.21.15.192:0/934429503 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 293673493 0 0) 0x7f205413eae0 con 0x7f20540b4b50 2024-03-25T19:14:07.856 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.852+0000 7f204bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:14:07.856 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.852+0000 7f204bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:07.856 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.852+0000 7f204bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:07.856 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.852+0000 7f204bfff640 10 monclient(hunting): my global_id is 4121 2024-03-25T19:14:07.856 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.852+0000 7f204bfff640 1 -- 172.21.15.192:0/934429503 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2034004eb0 con 0x7f20540b4b50 2024-03-25T19:14:07.856 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.852+0000 7f204bfff640 1 -- 172.21.15.192:0/934429503 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2957180820 0 0) 0x7f2054143b40 con 0x7f2054142dd0 2024-03-25T19:14:07.856 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.852+0000 7f204bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:14:07.856 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.852+0000 7f204bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:07.856 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.852+0000 7f204bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:07.856 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.852+0000 7f204bfff640 10 monclient(hunting): my global_id is 4123 2024-03-25T19:14:07.856 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.852+0000 7f204bfff640 1 -- 172.21.15.192:0/934429503 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f205413eae0 con 0x7f2054142dd0 2024-03-25T19:14:07.857 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.852+0000 7f204bfff640 1 -- 172.21.15.192:0/934429503 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3601067083 0 0) 0x7f20541439b0 con 0x7f205413ff20 2024-03-25T19:14:07.857 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.852+0000 7f204bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:14:07.857 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.852+0000 7f204bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:07.857 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.852+0000 7f204bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:07.857 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.852+0000 7f204bfff640 10 monclient(hunting): my global_id is 4125 2024-03-25T19:14:07.857 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.852+0000 7f204bfff640 1 -- 172.21.15.192:0/934429503 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2054143b40 con 0x7f205413ff20 2024-03-25T19:14:07.858 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.857+0000 7f204bfff640 1 -- 172.21.15.192:0/934429503 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3840517210 0 0) 0x7f205413eae0 con 0x7f2054142dd0 2024-03-25T19:14:07.858 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.857+0000 7f204bfff640 1 -- 172.21.15.192:0/934429503 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20541439b0 con 0x7f2054142dd0 2024-03-25T19:14:07.858 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.857+0000 7f204bfff640 1 -- 172.21.15.192:0/934429503 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f2044001b20 con 0x7f2054142dd0 2024-03-25T19:14:07.858 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.857+0000 7f204bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:07.858 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.857+0000 7f204bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:14:07.858 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.857+0000 7f204bfff640 10 monclient(hunting): dump: 2024-03-25T19:14:07.859 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:07.859 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:07.859 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:07.859 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:07.859 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:07.859 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:07.859 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:07.859 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:07.859 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:07.859 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:07.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.857+0000 7f204bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:14:07.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.858+0000 7f204bfff640 1 -- 172.21.15.192:0/934429503 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2324136289 0 0) 0x7f2054143b40 con 0x7f205413ff20 2024-03-25T19:14:07.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.858+0000 7f204bfff640 1 -- 172.21.15.192:0/934429503 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f205413eae0 con 0x7f205413ff20 2024-03-25T19:14:07.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.858+0000 7f204bfff640 1 -- 172.21.15.192:0/934429503 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f203c001910 con 0x7f205413ff20 2024-03-25T19:14:07.860 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.858+0000 7f204bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:07.860 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.858+0000 7f204bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:14:07.860 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.858+0000 7f204bfff640 10 monclient(hunting): dump: 2024-03-25T19:14:07.860 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:07.860 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:07.860 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:07.860 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:07.860 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:07.860 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:07.860 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:07.860 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:07.860 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:07.860 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:07.862 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.860+0000 7f204bfff640 1 -- 172.21.15.192:0/934429503 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1891489948 0 0) 0x7f20541439b0 con 0x7f2054142dd0 2024-03-25T19:14:07.862 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.860+0000 7f204bfff640 1 -- 172.21.15.192:0/934429503 >> v1:172.21.15.6:6790/0 conn(0x7f20540b4b50 legacy=0x7f205413a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:07.862 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.860+0000 7f204bfff640 1 -- 172.21.15.192:0/934429503 >> v1:172.21.15.6:6789/0 conn(0x7f205413ff20 legacy=0x7f2054139b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:07.862 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.860+0000 7f204bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:14:07.862 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.860+0000 7f204bfff640 1 monclient: found mon.b 2024-03-25T19:14:07.862 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.860+0000 7f204bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:14:07.862 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.860+0000 7f204bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:07.862 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.860+0000 7f204bfff640 1 -- 172.21.15.192:0/934429503 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f205413ec70 con 0x7f2054142dd0 2024-03-25T19:14:07.862 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.860+0000 7f204bfff640 10 monclient: _finish_auth 0 2024-03-25T19:14:07.862 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.860+0000 7f204bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:14:07.863 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.860+0000 7f204bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:14:07.863 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.860+0000 7f2059e4e640 5 monclient: authenticate success, global_id 4123 2024-03-25T19:14:07.863 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.860+0000 7f2059e4e640 10 monclient: _renew_subs 2024-03-25T19:14:07.863 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.860+0000 7f2059e4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:07.863 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.860+0000 7f2059e4e640 1 -- 172.21.15.192:0/934429503 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2054137f90 con 0x7f2054142dd0 2024-03-25T19:14:07.863 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.860+0000 7f2059e4e640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:14:07.863 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.861+0000 7f2059e4e640 10 monclient: _renew_subs 2024-03-25T19:14:07.863 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.861+0000 7f2059e4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:07.863 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.861+0000 7f2059e4e640 1 -- 172.21.15.192:0/934429503 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f20541e3b60 con 0x7f2054142dd0 2024-03-25T19:14:07.863 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.861+0000 7f2059e4e640 1 librados: init done 2024-03-25T19:14:07.863 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.861+0000 7f2059e4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:07.863 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.861+0000 7f2059e4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:07.863 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.861+0000 7f2059e4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:07.863 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.861+0000 7f2059e4e640 1 -- 172.21.15.192:0/934429503 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f205413a2a0 con 0x7f2054142dd0 2024-03-25T19:14:07.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.864+0000 7f204bfff640 1 -- 172.21.15.192:0/934429503 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2044001cc0 con 0x7f2054142dd0 2024-03-25T19:14:07.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.864+0000 7f204bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:14:07.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.864+0000 7f204bfff640 1 -- 172.21.15.192:0/934429503 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f2044003a40 con 0x7f2054142dd0 2024-03-25T19:14:07.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.864+0000 7f204bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:07.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.864+0000 7f204bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:14:07.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.864+0000 7f204bfff640 10 monclient: dump: 2024-03-25T19:14:07.865 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:07.865 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:07.865 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:07.865 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:07.865 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:07.865 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:07.866 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:07.866 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:07.866 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:07.866 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:07.866 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.864+0000 7f204bfff640 1 -- 172.21.15.192:0/934429503 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2044003f30 con 0x7f2054142dd0 2024-03-25T19:14:07.866 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.864+0000 7f2051ffb640 4 set_mon_vals no callback set 2024-03-25T19:14:07.866 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.864+0000 7f204bfff640 1 -- 172.21.15.192:0/934429503 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 2203492735 0 0) 0x7f2044005470 con 0x7f2054142dd0 2024-03-25T19:14:07.866 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.864+0000 7f204bfff640 10 client.4123.objecter ms_dispatch 0x7f2054000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T19:14:07.866 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.864+0000 7f204bfff640 3 client.4123.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T19:14:07.866 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.864+0000 7f204bfff640 3 client.4123.objecter handle_osd_map decoding full epoch 3 2024-03-25T19:14:07.866 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.864+0000 7f204bfff640 20 client.4123.objecter dump_active .. 0 homeless 2024-03-25T19:14:07.956 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.956+0000 7f204bfff640 1 -- 172.21.15.192:0/934429503 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f20440057d0 con 0x7f2054142dd0 2024-03-25T19:14:07.956 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.956+0000 7f204bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:07.956 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:07.956+0000 7f204bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:14:08.028 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.028+0000 7f2059e4e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "2a83e0b2-dbc3-40be-8026-48e14034e957", "id": 1}] 2024-03-25T19:14:08.029 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.028+0000 7f2059e4e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "2a83e0b2-dbc3-40be-8026-48e14034e957", "id": 1}] 2024-03-25T19:14:08.029 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.028+0000 7f2059e4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:08.029 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.028+0000 7f2059e4e640 1 -- 172.21.15.192:0/934429503 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd new", "uuid": "2a83e0b2-dbc3-40be-8026-48e14034e957", "id": 1} v 0) v1 -- 0x7f205413ee60 con 0x7f2054142dd0 2024-03-25T19:14:08.122 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.122+0000 7f204bfff640 1 -- 172.21.15.192:0/934429503 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2a83e0b2-dbc3-40be-8026-48e14034e957", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 2151754390 0 940483276) 0x7f2044005a10 con 0x7f2054142dd0 2024-03-25T19:14:08.122 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.122+0000 7f204bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "2a83e0b2-dbc3-40be-8026-48e14034e957", "id": 1}] 2024-03-25T19:14:08.122 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.122+0000 7f204bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:14:08.122 INFO:teuthology.orchestra.run.smithi192.stdout:1 2024-03-25T19:14:08.124 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.123+0000 7f2059e4e640 10 librados: watch_flush enter 2024-03-25T19:14:08.124 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.123+0000 7f2059e4e640 10 librados: watch_flush exit 2024-03-25T19:14:08.124 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.123+0000 7f2059e4e640 20 client.4123.objecter shutdown clearing up homeless session... 2024-03-25T19:14:08.124 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.123+0000 7f2059e4e640 10 client.4123.objecter successfully canceled tick 2024-03-25T19:14:08.124 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.123+0000 7f2059e4e640 10 monclient: shutdown 2024-03-25T19:14:08.124 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.123+0000 7f2059e4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:14:08.124 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.123+0000 7f2059e4e640 1 -- 172.21.15.192:0/934429503 >> v1:172.21.15.192:6789/0 conn(0x7f2054142dd0 legacy=0x7f20541e2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:08.124 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.123+0000 7f2059e4e640 1 -- 172.21.15.192:0/934429503 shutdown_connections 2024-03-25T19:14:08.124 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.123+0000 7f2059e4e640 1 -- 172.21.15.192:0/934429503 >> 172.21.15.192:0/934429503 conn(0x7f2054066920 msgr2=0x7f20540b6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:14:08.124 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.123+0000 7f2059e4e640 1 -- 172.21.15.192:0/934429503 shutdown_connections 2024-03-25T19:14:08.124 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.123+0000 7f2059e4e640 1 -- 172.21.15.192:0/934429503 wait complete. 2024-03-25T19:14:08.124 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.123+0000 7f2059e4e640 1 librados: shutdown 2024-03-25T19:14:08.134 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new 3072cff1-2822-4922-9ec1-97b6e8656a3a 2 2024-03-25T19:14:08.205 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-25T19:14:08.206+0000 7fbad1252040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T19:14:08.234 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.234+0000 7f2d371d6640 10 monclient: get_monmap_and_config 2024-03-25T19:14:08.234 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.234+0000 7f2d371d6640 10 monclient: build_initial_monmap 2024-03-25T19:14:08.234 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.234+0000 7f2d371d6640 10 monclient: monmap: 2024-03-25T19:14:08.234 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2024-03-25T19:14:08.234 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:08.234 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:14:08.235149+0000 2024-03-25T19:14:08.234 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:14:08.235149+0000 2024-03-25T19:14:08.234 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2024-03-25T19:14:08.234 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:08.234 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:14:08.234 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:14:08.235 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:14:08.235 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:08.236 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.234+0000 7f2d371d6640 1 Processor -- start 2024-03-25T19:14:08.236 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.234+0000 7f2d371d6640 1 -- start start 2024-03-25T19:14:08.236 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.234+0000 7f2d371d6640 10 monclient: init 2024-03-25T19:14:08.236 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.235+0000 7f2d371d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:14:08.236 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.235+0000 7f2d371d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:14:08.236 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.235+0000 7f2d371d6640 10 monclient(hunting): picked mon.noname-a con 0x7f2d3006f300 addr v1:172.21.15.6:6789/0 2024-03-25T19:14:08.236 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.235+0000 7f2d371d6640 10 monclient(hunting): picked mon.noname-b con 0x7f2d3013f180 addr v1:172.21.15.6:6790/0 2024-03-25T19:14:08.237 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.235+0000 7f2d371d6640 10 monclient(hunting): picked mon.noname-c con 0x7f2d3013f810 addr v1:172.21.15.192:6789/0 2024-03-25T19:14:08.237 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.235+0000 7f2d371d6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d30072c40 con 0x7f2d3006f300 2024-03-25T19:14:08.237 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.235+0000 7f2d371d6640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d3013f5e0 con 0x7f2d3013f810 2024-03-25T19:14:08.237 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.235+0000 7f2d371d6640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d30071830 con 0x7f2d3013f180 2024-03-25T19:14:08.237 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.235+0000 7f2d371d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:14:08.237 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.235+0000 7f2d371d6640 10 monclient(hunting): authenticate will time out at 5970.529023s 2024-03-25T19:14:08.237 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.236+0000 7f2d34f72640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2d3006f300 0x7f2d3006f6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.192:42808/0 (socket says 172.21.15.192:42808) 2024-03-25T19:14:08.237 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.236+0000 7f2d34f72640 1 -- 172.21.15.192:0/2950015950 learned_addr learned my addr 172.21.15.192:0/2950015950 (peer_addr_for_me v1:172.21.15.192:0/0) 2024-03-25T19:14:08.239 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.238+0000 7f2d2f7fe640 1 -- 172.21.15.192:0/2950015950 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3678966483 0 0) 0x7f2d3013f5e0 con 0x7f2d3013f810 2024-03-25T19:14:08.239 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.238+0000 7f2d2f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:14:08.239 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.238+0000 7f2d2f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:08.239 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.238+0000 7f2d2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:08.239 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.238+0000 7f2d2f7fe640 10 monclient(hunting): my global_id is 4126 2024-03-25T19:14:08.239 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.238+0000 7f2d2f7fe640 1 -- 172.21.15.192:0/2950015950 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d20003520 con 0x7f2d3013f810 2024-03-25T19:14:08.239 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.238+0000 7f2d2f7fe640 1 -- 172.21.15.192:0/2950015950 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1332627043 0 0) 0x7f2d30071830 con 0x7f2d3013f180 2024-03-25T19:14:08.239 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.238+0000 7f2d2f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:14:08.239 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.238+0000 7f2d2f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:08.239 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.238+0000 7f2d2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:08.239 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.238+0000 7f2d2f7fe640 10 monclient(hunting): my global_id is 4124 2024-03-25T19:14:08.239 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.238+0000 7f2d2f7fe640 1 -- 172.21.15.192:0/2950015950 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d3013f5e0 con 0x7f2d3013f180 2024-03-25T19:14:08.240 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.239+0000 7f2d2f7fe640 1 -- 172.21.15.192:0/2950015950 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3726182954 0 0) 0x7f2d30072c40 con 0x7f2d3006f300 2024-03-25T19:14:08.240 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.239+0000 7f2d2f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:14:08.240 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.239+0000 7f2d2f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:08.240 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.239+0000 7f2d2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:08.240 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.239+0000 7f2d2f7fe640 10 monclient(hunting): my global_id is 4128 2024-03-25T19:14:08.240 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.239+0000 7f2d2f7fe640 1 -- 172.21.15.192:0/2950015950 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d30071830 con 0x7f2d3006f300 2024-03-25T19:14:08.240 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-03-25T19:14:08.240+0000 7f1c54213040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T19:14:08.242 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.241+0000 7f2d2f7fe640 1 -- 172.21.15.192:0/2950015950 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2982407626 0 0) 0x7f2d20003520 con 0x7f2d3013f810 2024-03-25T19:14:08.242 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.241+0000 7f2d2f7fe640 1 -- 172.21.15.192:0/2950015950 >> v1:172.21.15.6:6790/0 conn(0x7f2d3013f180 legacy=0x7f2d30072530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:08.242 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.241+0000 7f2d2f7fe640 1 -- 172.21.15.192:0/2950015950 >> v1:172.21.15.6:6789/0 conn(0x7f2d3006f300 legacy=0x7f2d3006f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:08.242 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.241+0000 7f2d2f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:14:08.242 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.241+0000 7f2d2f7fe640 1 monclient: found mon.noname-c 2024-03-25T19:14:08.242 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.241+0000 7f2d2f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:14:08.242 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.241+0000 7f2d2f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:14:08.242 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.241+0000 7f2d2f7fe640 1 -- 172.21.15.192:0/2950015950 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d30071b10 con 0x7f2d3013f810 2024-03-25T19:14:08.242 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.241+0000 7f2d2f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:14:08.243 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.241+0000 7f2d2f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:14:08.243 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.241+0000 7f2d2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:14:08.243 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.241+0000 7f2d2f7fe640 1 -- 172.21.15.192:0/2950015950 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f2d28001a10 con 0x7f2d3013f810 2024-03-25T19:14:08.243 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.241+0000 7f2d2f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:08.243 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.241+0000 7f2d2f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:14:08.243 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.241+0000 7f2d2f7fe640 10 monclient: dump: 2024-03-25T19:14:08.243 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:08.243 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:08.243 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:08.243 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:08.243 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:08.243 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:08.243 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:08.243 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:08.244 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:08.244 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:08.244 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.241+0000 7f2d371d6640 5 monclient: authenticate success, global_id 4126 2024-03-25T19:14:08.244 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.241+0000 7f2d371d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:14:08.247 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.243+0000 7f2d2f7fe640 1 -- 172.21.15.192:0/2950015950 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d28001b70 con 0x7f2d3013f810 2024-03-25T19:14:08.247 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.243+0000 7f2d2f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:14:08.247 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.243+0000 7f2d2f7fe640 1 -- 172.21.15.192:0/2950015950 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f2d28001f80 con 0x7f2d3013f810 2024-03-25T19:14:08.247 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.243+0000 7f2d371d6640 10 monclient: get_monmap_and_config success 2024-03-25T19:14:08.247 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.243+0000 7f2d371d6640 4 set_mon_vals no callback set 2024-03-25T19:14:08.247 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.243+0000 7f2d2f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:08.247 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.243+0000 7f2d2f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:14:08.247 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.243+0000 7f2d2f7fe640 10 monclient: dump: 2024-03-25T19:14:08.247 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:08.247 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:08.247 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:08.247 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:08.247 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:08.247 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:08.248 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:08.248 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:08.248 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:08.248 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:08.248 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.244+0000 7f2d371d6640 10 monclient: shutdown 2024-03-25T19:14:08.248 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.244+0000 7f2d371d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:14:08.248 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.244+0000 7f2d371d6640 1 -- 172.21.15.192:0/2950015950 >> v1:172.21.15.192:6789/0 conn(0x7f2d3013f810 legacy=0x7f2d3014a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:08.248 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.244+0000 7f2d371d6640 1 -- 172.21.15.192:0/2950015950 shutdown_connections 2024-03-25T19:14:08.248 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.244+0000 7f2d371d6640 1 -- 172.21.15.192:0/2950015950 >> 172.21.15.192:0/2950015950 conn(0x7f2d300b4bc0 msgr2=0x7f2d300b4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:14:08.248 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.244+0000 7f2d371d6640 1 -- 172.21.15.192:0/2950015950 shutdown_connections 2024-03-25T19:14:08.248 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.244+0000 7f2d371d6640 1 -- 172.21.15.192:0/2950015950 wait complete. 2024-03-25T19:14:08.248 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.244+0000 7f2d371d6640 10 monclient: build_initial_monmap 2024-03-25T19:14:08.248 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.244+0000 7f2d371d6640 10 monclient: monmap: 2024-03-25T19:14:08.248 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2024-03-25T19:14:08.248 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:14:08.249 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 0.000000 2024-03-25T19:14:08.249 INFO:teuthology.orchestra.run.smithi192.stderr:created 0.000000 2024-03-25T19:14:08.249 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2024-03-25T19:14:08.249 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2024-03-25T19:14:08.249 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:14:08.249 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:14:08.249 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:14:08.249 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:08.249 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.245+0000 7f2d371d6640 1 librados: starting msgr at 2024-03-25T19:14:08.249 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.245+0000 7f2d371d6640 1 librados: starting objecter 2024-03-25T19:14:08.249 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.245+0000 7f2d371d6640 1 Processor -- start 2024-03-25T19:14:08.249 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.245+0000 7f2d371d6640 1 -- start start 2024-03-25T19:14:08.249 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.245+0000 7f2d371d6640 1 librados: setting wanted keys 2024-03-25T19:14:08.249 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.245+0000 7f2d371d6640 1 librados: calling monclient init 2024-03-25T19:14:08.249 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.245+0000 7f2d371d6640 10 monclient: init 2024-03-25T19:14:08.250 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.245+0000 7f2d371d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:14:08.250 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.245+0000 7f2d371d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:14:08.250 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.245+0000 7f2d371d6640 10 monclient(hunting): picked mon.noname-a con 0x7f2d3006f300 addr v1:172.21.15.6:6789/0 2024-03-25T19:14:08.250 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.245+0000 7f2d371d6640 10 monclient(hunting): picked mon.noname-b con 0x7f2d3013f180 addr v1:172.21.15.192:6789/0 2024-03-25T19:14:08.250 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.245+0000 7f2d371d6640 10 monclient(hunting): picked mon.noname-c con 0x7f2d3013f810 addr v1:172.21.15.6:6790/0 2024-03-25T19:14:08.250 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.246+0000 7f2d371d6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d301d7eb0 con 0x7f2d3006f300 2024-03-25T19:14:08.250 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.246+0000 7f2d371d6640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d30139990 con 0x7f2d3013f180 2024-03-25T19:14:08.250 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.246+0000 7f2d371d6640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d30122f40 con 0x7f2d3013f810 2024-03-25T19:14:08.250 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.246+0000 7f2d371d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:14:08.250 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.246+0000 7f2d371d6640 10 monclient(hunting): authenticate will time out at 5970.539414s 2024-03-25T19:14:08.250 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.246+0000 7f2d2ffff640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f2d3013f180 0x7f2d30138b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.192:58364/0 (socket says 172.21.15.192:58364) 2024-03-25T19:14:08.250 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.246+0000 7f2d2ffff640 1 -- 172.21.15.192:0/99121837 learned_addr learned my addr 172.21.15.192:0/99121837 (peer_addr_for_me v1:172.21.15.192:0/0) 2024-03-25T19:14:08.250 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.246+0000 7f2d2d7fa640 10 client.?.objecter ms_handle_connect 0x7f2d3013f180 2024-03-25T19:14:08.250 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.246+0000 7f2d2d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:08.250 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.246+0000 7f2d2d7fa640 10 client.?.objecter ms_handle_connect 0x7f2d3013f810 2024-03-25T19:14:08.251 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.246+0000 7f2d2d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:08.251 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.248+0000 7f2d2d7fa640 1 -- 172.21.15.192:0/99121837 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1199472483 0 0) 0x7f2d30139990 con 0x7f2d3013f180 2024-03-25T19:14:08.251 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.248+0000 7f2d2d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:14:08.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.248+0000 7f2d2d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:08.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.248+0000 7f2d2d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:08.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.248+0000 7f2d2d7fa640 10 monclient(hunting): my global_id is 4129 2024-03-25T19:14:08.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.248+0000 7f2d2d7fa640 1 -- 172.21.15.192:0/99121837 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d14004eb0 con 0x7f2d3013f180 2024-03-25T19:14:08.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.249+0000 7f2d2d7fa640 1 -- 172.21.15.192:0/99121837 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2631886648 0 0) 0x7f2d30122f40 con 0x7f2d3013f810 2024-03-25T19:14:08.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.249+0000 7f2d2d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:14:08.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.249+0000 7f2d2d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:08.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.249+0000 7f2d2d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:08.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.249+0000 7f2d2d7fa640 10 monclient(hunting): my global_id is 4127 2024-03-25T19:14:08.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.249+0000 7f2d2d7fa640 1 -- 172.21.15.192:0/99121837 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d30139990 con 0x7f2d3013f810 2024-03-25T19:14:08.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.249+0000 7f2d2d7fa640 10 client.?.objecter ms_handle_connect 0x7f2d3006f300 2024-03-25T19:14:08.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.249+0000 7f2d2d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:08.253 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.252+0000 7f2d2d7fa640 1 -- 172.21.15.192:0/99121837 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 113915822 0 0) 0x7f2d301d7eb0 con 0x7f2d3006f300 2024-03-25T19:14:08.253 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.252+0000 7f2d2d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:14:08.253 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.252+0000 7f2d2d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:08.253 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.252+0000 7f2d2d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:08.253 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.252+0000 7f2d2d7fa640 10 monclient(hunting): my global_id is 4131 2024-03-25T19:14:08.253 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.252+0000 7f2d2d7fa640 1 -- 172.21.15.192:0/99121837 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d30122f40 con 0x7f2d3006f300 2024-03-25T19:14:08.254 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.253+0000 7f2d2d7fa640 1 -- 172.21.15.192:0/99121837 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3090895077 0 0) 0x7f2d14004eb0 con 0x7f2d3013f180 2024-03-25T19:14:08.254 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.253+0000 7f2d2d7fa640 1 -- 172.21.15.192:0/99121837 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d301d7eb0 con 0x7f2d3013f180 2024-03-25T19:14:08.254 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.253+0000 7f2d2d7fa640 1 -- 172.21.15.192:0/99121837 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f2d1c001a30 con 0x7f2d3013f180 2024-03-25T19:14:08.254 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.253+0000 7f2d2d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:08.254 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.253+0000 7f2d2d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:14:08.255 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.253+0000 7f2d2d7fa640 10 monclient(hunting): dump: 2024-03-25T19:14:08.255 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:08.255 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:08.255 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:08.255 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:08.255 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:08.255 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:08.255 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:08.255 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:08.255 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:08.255 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:08.255 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.253+0000 7f2d2d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:14:08.255 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.254+0000 7f2d2d7fa640 1 -- 172.21.15.192:0/99121837 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 971555097 0 0) 0x7f2d30139990 con 0x7f2d3013f810 2024-03-25T19:14:08.255 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.254+0000 7f2d2d7fa640 1 -- 172.21.15.192:0/99121837 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d14004eb0 con 0x7f2d3013f810 2024-03-25T19:14:08.255 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.254+0000 7f2d2d7fa640 1 -- 172.21.15.192:0/99121837 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f2d28001e80 con 0x7f2d3013f810 2024-03-25T19:14:08.256 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.254+0000 7f2d2d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:08.256 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.254+0000 7f2d2d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:14:08.256 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.254+0000 7f2d2d7fa640 10 monclient(hunting): dump: 2024-03-25T19:14:08.256 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:08.256 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:08.256 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:08.256 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:08.256 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:08.256 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:08.256 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:08.256 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:08.256 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:08.256 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:08.257 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.257+0000 7f2d2d7fa640 1 -- 172.21.15.192:0/99121837 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2836513868 0 0) 0x7f2d301d7eb0 con 0x7f2d3013f180 2024-03-25T19:14:08.257 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.257+0000 7f2d2d7fa640 1 -- 172.21.15.192:0/99121837 >> v1:172.21.15.6:6790/0 conn(0x7f2d3013f810 legacy=0x7f2d30139280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:08.257 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.257+0000 7f2d2d7fa640 1 -- 172.21.15.192:0/99121837 >> v1:172.21.15.6:6789/0 conn(0x7f2d3006f300 legacy=0x7f2d30120ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:08.257 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.257+0000 7f2d2d7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:14:08.258 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.257+0000 7f2d2d7fa640 1 monclient: found mon.b 2024-03-25T19:14:08.258 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.257+0000 7f2d2d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:14:08.258 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.257+0000 7f2d2d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:08.258 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.257+0000 7f2d2d7fa640 1 -- 172.21.15.192:0/99121837 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d30123220 con 0x7f2d3013f180 2024-03-25T19:14:08.258 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.257+0000 7f2d2d7fa640 10 monclient: _finish_auth 0 2024-03-25T19:14:08.258 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.257+0000 7f2d2d7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:14:08.258 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.257+0000 7f2d2d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:14:08.258 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.257+0000 7f2d371d6640 5 monclient: authenticate success, global_id 4129 2024-03-25T19:14:08.258 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.257+0000 7f2d371d6640 10 monclient: _renew_subs 2024-03-25T19:14:08.258 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.257+0000 7f2d371d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:08.258 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.257+0000 7f2d371d6640 1 -- 172.21.15.192:0/99121837 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d30139bb0 con 0x7f2d3013f180 2024-03-25T19:14:08.258 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.257+0000 7f2d371d6640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:14:08.258 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.257+0000 7f2d371d6640 10 monclient: _renew_subs 2024-03-25T19:14:08.258 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.257+0000 7f2d371d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:08.259 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.257+0000 7f2d371d6640 1 -- 172.21.15.192:0/99121837 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d301215b0 con 0x7f2d3013f180 2024-03-25T19:14:08.259 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.257+0000 7f2d371d6640 1 librados: init done 2024-03-25T19:14:08.259 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.258+0000 7f2d371d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:08.259 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.258+0000 7f2d371d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:08.259 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.258+0000 7f2d371d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:08.259 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.258+0000 7f2d371d6640 1 -- 172.21.15.192:0/99121837 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d301d8060 con 0x7f2d3013f180 2024-03-25T19:14:08.261 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.261+0000 7f2d2d7fa640 1 -- 172.21.15.192:0/99121837 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d1c001c70 con 0x7f2d3013f180 2024-03-25T19:14:08.261 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.261+0000 7f2d2d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:14:08.261 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.261+0000 7f2d2d7fa640 1 -- 172.21.15.192:0/99121837 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f2d1c0038c0 con 0x7f2d3013f180 2024-03-25T19:14:08.261 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.261+0000 7f2d2d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:08.261 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.261+0000 7f2d2d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:14:08.262 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.261+0000 7f2d2d7fa640 10 monclient: dump: 2024-03-25T19:14:08.262 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:08.262 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:08.262 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:08.262 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:08.262 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:08.262 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:08.262 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:08.262 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:08.262 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:08.262 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:08.262 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.261+0000 7f2d2effd640 4 set_mon_vals no callback set 2024-03-25T19:14:08.262 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.261+0000 7f2d2d7fa640 1 -- 172.21.15.192:0/99121837 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2d1c003e70 con 0x7f2d3013f180 2024-03-25T19:14:08.262 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.261+0000 7f2d2d7fa640 1 -- 172.21.15.192:0/99121837 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 2819767361 0 0) 0x7f2d1c006420 con 0x7f2d3013f180 2024-03-25T19:14:08.262 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.261+0000 7f2d2d7fa640 10 client.4129.objecter ms_dispatch 0x7f2d30000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T19:14:08.262 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.261+0000 7f2d2d7fa640 3 client.4129.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T19:14:08.263 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.261+0000 7f2d2d7fa640 3 client.4129.objecter handle_osd_map decoding full epoch 4 2024-03-25T19:14:08.263 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.261+0000 7f2d2d7fa640 20 client.4129.objecter dump_active .. 0 homeless 2024-03-25T19:14:08.350 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-25T19:14:08.350+0000 7fbad1252040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T19:14:08.353 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.353+0000 7f2d2d7fa640 1 -- 172.21.15.192:0/99121837 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2d1c006110 con 0x7f2d3013f180 2024-03-25T19:14:08.353 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.353+0000 7f2d2d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:08.353 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.353+0000 7f2d2d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:14:08.378 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-03-25T19:14:08.378+0000 7f1c54213040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T19:14:08.426 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.425+0000 7f2d371d6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "3072cff1-2822-4922-9ec1-97b6e8656a3a", "id": 2}] 2024-03-25T19:14:08.426 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.425+0000 7f2d371d6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "3072cff1-2822-4922-9ec1-97b6e8656a3a", "id": 2}] 2024-03-25T19:14:08.426 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.426+0000 7f2d371d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:08.426 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.426+0000 7f2d371d6640 1 -- 172.21.15.192:0/99121837 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd new", "uuid": "3072cff1-2822-4922-9ec1-97b6e8656a3a", "id": 2} v 0) v1 -- 0x7f2d3004f730 con 0x7f2d3013f180 2024-03-25T19:14:08.481 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-25T19:14:08.481+0000 7fbad1252040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T19:14:08.493 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.493+0000 7f2d2d7fa640 1 -- 172.21.15.192:0/99121837 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3072cff1-2822-4922-9ec1-97b6e8656a3a", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 2804916584 0 216599125) 0x7f2d1c0057c0 con 0x7f2d3013f180 2024-03-25T19:14:08.493 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.493+0000 7f2d2d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "3072cff1-2822-4922-9ec1-97b6e8656a3a", "id": 2}] 2024-03-25T19:14:08.493 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.493+0000 7f2d2d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:14:08.493 INFO:teuthology.orchestra.run.smithi192.stdout:2 2024-03-25T19:14:08.494 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.493+0000 7f2d371d6640 10 librados: watch_flush enter 2024-03-25T19:14:08.495 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.493+0000 7f2d371d6640 10 librados: watch_flush exit 2024-03-25T19:14:08.495 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.493+0000 7f2d371d6640 20 client.4129.objecter shutdown clearing up homeless session... 2024-03-25T19:14:08.495 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.493+0000 7f2d371d6640 10 client.4129.objecter successfully canceled tick 2024-03-25T19:14:08.495 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.494+0000 7f2d371d6640 10 monclient: shutdown 2024-03-25T19:14:08.495 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.494+0000 7f2d371d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:14:08.495 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.494+0000 7f2d371d6640 1 -- 172.21.15.192:0/99121837 >> v1:172.21.15.192:6789/0 conn(0x7f2d3013f180 legacy=0x7f2d30138b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:08.495 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.494+0000 7f2d371d6640 1 -- 172.21.15.192:0/99121837 shutdown_connections 2024-03-25T19:14:08.495 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.494+0000 7f2d371d6640 1 -- 172.21.15.192:0/99121837 >> 172.21.15.192:0/99121837 conn(0x7f2d300b4bc0 msgr2=0x7f2d30074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:14:08.495 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.494+0000 7f2d371d6640 1 -- 172.21.15.192:0/99121837 shutdown_connections 2024-03-25T19:14:08.495 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.494+0000 7f2d371d6640 1 -- 172.21.15.192:0/99121837 wait complete. 2024-03-25T19:14:08.495 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.494+0000 7f2d371d6640 1 librados: shutdown 2024-03-25T19:14:08.504 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new d14827e0-9625-4ef5-8adf-d3430bde36e7 3 2024-03-25T19:14:08.531 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-25T19:14:08.531+0000 7fbad1252040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T19:14:08.601 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.601+0000 7fb802d23640 10 monclient: get_monmap_and_config 2024-03-25T19:14:08.602 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.601+0000 7fb802d23640 10 monclient: build_initial_monmap 2024-03-25T19:14:08.602 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.601+0000 7fb802d23640 10 monclient: monmap: 2024-03-25T19:14:08.602 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2024-03-25T19:14:08.602 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:08.602 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:14:08.602775+0000 2024-03-25T19:14:08.602 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:14:08.602775+0000 2024-03-25T19:14:08.602 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2024-03-25T19:14:08.602 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:08.602 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:14:08.602 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:14:08.602 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:14:08.602 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:08.604 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.602+0000 7fb802d23640 1 Processor -- start 2024-03-25T19:14:08.604 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.602+0000 7fb802d23640 1 -- start start 2024-03-25T19:14:08.604 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.602+0000 7fb802d23640 10 monclient: init 2024-03-25T19:14:08.604 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.603+0000 7fb802d23640 10 monclient: _reopen_session rank -1 2024-03-25T19:14:08.604 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.603+0000 7fb802d23640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:14:08.604 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.603+0000 7fb802d23640 10 monclient(hunting): picked mon.noname-c con 0x7fb7fc0aa210 addr v1:172.21.15.192:6789/0 2024-03-25T19:14:08.604 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.603+0000 7fb802d23640 10 monclient(hunting): picked mon.noname-b con 0x7fb7fc0ad270 addr v1:172.21.15.6:6790/0 2024-03-25T19:14:08.604 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.603+0000 7fb802d23640 10 monclient(hunting): picked mon.noname-a con 0x7fb7fc0ab190 addr v1:172.21.15.6:6789/0 2024-03-25T19:14:08.605 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.603+0000 7fb802d23640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7fc0ac180 con 0x7fb7fc0ab190 2024-03-25T19:14:08.605 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.603+0000 7fb802d23640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7fc0ac460 con 0x7fb7fc0aa210 2024-03-25T19:14:08.605 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.603+0000 7fb802d23640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7fc0ac740 con 0x7fb7fc0ad270 2024-03-25T19:14:08.605 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.603+0000 7fb802d23640 10 monclient(hunting): _renew_subs 2024-03-25T19:14:08.605 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.603+0000 7fb802d23640 10 monclient(hunting): authenticate will time out at 5970.896677s 2024-03-25T19:14:08.605 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.603+0000 7fb8012c0640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb7fc0ab190 0x7fb7fc14a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.192:42824/0 (socket says 172.21.15.192:42824) 2024-03-25T19:14:08.605 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.603+0000 7fb8012c0640 1 -- 172.21.15.192:0/2863594133 learned_addr learned my addr 172.21.15.192:0/2863594133 (peer_addr_for_me v1:172.21.15.192:0/0) 2024-03-25T19:14:08.605 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.603+0000 7fb800abf640 1 --1- 172.21.15.192:0/2863594133 >> v1:172.21.15.192:6789/0 conn(0x7fb7fc0aa210 0x7fb7fc0acb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.192:58378/0 (socket says 172.21.15.192:58378) 2024-03-25T19:14:08.606 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.606+0000 7fb7fb7fe640 1 -- 172.21.15.192:0/2863594133 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 400569677 0 0) 0x7fb7fc0ac460 con 0x7fb7fc0aa210 2024-03-25T19:14:08.606 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.606+0000 7fb7fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:14:08.606 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.606+0000 7fb7fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:08.606 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.606+0000 7fb7fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:08.607 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.606+0000 7fb7fb7fe640 10 monclient(hunting): my global_id is 4132 2024-03-25T19:14:08.607 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.606+0000 7fb7fb7fe640 1 -- 172.21.15.192:0/2863594133 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7e4003520 con 0x7fb7fc0aa210 2024-03-25T19:14:08.607 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.606+0000 7fb7fb7fe640 1 -- 172.21.15.192:0/2863594133 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 635760182 0 0) 0x7fb7fc0ac740 con 0x7fb7fc0ad270 2024-03-25T19:14:08.607 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.606+0000 7fb7fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:14:08.607 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.606+0000 7fb7fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:08.607 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.606+0000 7fb7fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:08.607 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.606+0000 7fb7fb7fe640 10 monclient(hunting): my global_id is 4130 2024-03-25T19:14:08.607 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.606+0000 7fb7fb7fe640 1 -- 172.21.15.192:0/2863594133 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7fc0ac460 con 0x7fb7fc0ad270 2024-03-25T19:14:08.607 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.606+0000 7fb7fb7fe640 1 -- 172.21.15.192:0/2863594133 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 999016987 0 0) 0x7fb7fc0ac180 con 0x7fb7fc0ab190 2024-03-25T19:14:08.607 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.606+0000 7fb7fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:14:08.607 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.606+0000 7fb7fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:08.607 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.606+0000 7fb7fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:08.607 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.606+0000 7fb7fb7fe640 10 monclient(hunting): my global_id is 4134 2024-03-25T19:14:08.607 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.606+0000 7fb7fb7fe640 1 -- 172.21.15.192:0/2863594133 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7fc0ac740 con 0x7fb7fc0ab190 2024-03-25T19:14:08.609 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.609+0000 7fb7fb7fe640 1 -- 172.21.15.192:0/2863594133 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 124545796 0 0) 0x7fb7e4003520 con 0x7fb7fc0aa210 2024-03-25T19:14:08.609 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.609+0000 7fb7fb7fe640 1 -- 172.21.15.192:0/2863594133 >> v1:172.21.15.6:6790/0 conn(0x7fb7fc0ad270 legacy=0x7fb7fc13ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:08.609 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.609+0000 7fb7fb7fe640 1 -- 172.21.15.192:0/2863594133 >> v1:172.21.15.6:6789/0 conn(0x7fb7fc0ab190 legacy=0x7fb7fc14a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:08.609 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.609+0000 7fb7fb7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:14:08.609 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.609+0000 7fb7fb7fe640 1 monclient: found mon.noname-c 2024-03-25T19:14:08.609 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.609+0000 7fb7fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:14:08.610 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.609+0000 7fb7fb7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:14:08.610 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.609+0000 7fb7fb7fe640 1 -- 172.21.15.192:0/2863594133 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7fc0aac70 con 0x7fb7fc0aa210 2024-03-25T19:14:08.610 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.609+0000 7fb7fb7fe640 10 monclient: _finish_auth 0 2024-03-25T19:14:08.610 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.609+0000 7fb7fb7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:14:08.610 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.609+0000 7fb7fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:14:08.610 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.609+0000 7fb802d23640 5 monclient: authenticate success, global_id 4132 2024-03-25T19:14:08.610 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.609+0000 7fb7fb7fe640 1 -- 172.21.15.192:0/2863594133 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fb7ec001690 con 0x7fb7fc0aa210 2024-03-25T19:14:08.610 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.609+0000 7fb802d23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:14:08.610 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.609+0000 7fb7fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:08.610 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.609+0000 7fb7fb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:14:08.610 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.609+0000 7fb7fb7fe640 10 monclient: dump: 2024-03-25T19:14:08.610 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:08.610 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:08.611 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:08.611 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:08.611 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:08.611 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:08.611 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:08.611 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:08.611 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:08.611 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:08.611 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.609+0000 7fb802d23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:14:08.614 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.610+0000 7fb7fb7fe640 1 -- 172.21.15.192:0/2863594133 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7ec0017f0 con 0x7fb7fc0aa210 2024-03-25T19:14:08.614 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.610+0000 7fb7fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:14:08.614 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.610+0000 7fb7fb7fe640 1 -- 172.21.15.192:0/2863594133 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fb7ec001c00 con 0x7fb7fc0aa210 2024-03-25T19:14:08.614 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.610+0000 7fb802d23640 10 monclient: get_monmap_and_config success 2024-03-25T19:14:08.614 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.610+0000 7fb802d23640 4 set_mon_vals no callback set 2024-03-25T19:14:08.614 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.610+0000 7fb7fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:08.614 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.610+0000 7fb7fb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:14:08.615 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.610+0000 7fb7fb7fe640 10 monclient: dump: 2024-03-25T19:14:08.615 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:08.615 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:08.615 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:08.615 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:08.615 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:08.615 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:08.615 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:08.615 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:08.615 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:08.615 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:08.615 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.611+0000 7fb802d23640 10 monclient: shutdown 2024-03-25T19:14:08.615 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.611+0000 7fb802d23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:14:08.615 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.611+0000 7fb802d23640 1 -- 172.21.15.192:0/2863594133 >> v1:172.21.15.192:6789/0 conn(0x7fb7fc0aa210 legacy=0x7fb7fc0acb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:08.616 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.611+0000 7fb802d23640 1 -- 172.21.15.192:0/2863594133 shutdown_connections 2024-03-25T19:14:08.616 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.611+0000 7fb802d23640 1 -- 172.21.15.192:0/2863594133 >> 172.21.15.192:0/2863594133 conn(0x7fb7fc0b3b20 msgr2=0x7fb7fc0b3f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:14:08.616 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.611+0000 7fb802d23640 1 -- 172.21.15.192:0/2863594133 shutdown_connections 2024-03-25T19:14:08.616 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.611+0000 7fb802d23640 1 -- 172.21.15.192:0/2863594133 wait complete. 2024-03-25T19:14:08.616 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.611+0000 7fb802d23640 10 monclient: build_initial_monmap 2024-03-25T19:14:08.616 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.611+0000 7fb802d23640 10 monclient: monmap: 2024-03-25T19:14:08.616 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2024-03-25T19:14:08.616 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:14:08.616 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 0.000000 2024-03-25T19:14:08.616 INFO:teuthology.orchestra.run.smithi192.stderr:created 0.000000 2024-03-25T19:14:08.616 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2024-03-25T19:14:08.616 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2024-03-25T19:14:08.616 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:14:08.616 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:14:08.616 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:14:08.617 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:08.617 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.612+0000 7fb802d23640 1 librados: starting msgr at 2024-03-25T19:14:08.617 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.612+0000 7fb802d23640 1 librados: starting objecter 2024-03-25T19:14:08.617 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.612+0000 7fb802d23640 1 Processor -- start 2024-03-25T19:14:08.617 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.612+0000 7fb802d23640 1 -- start start 2024-03-25T19:14:08.617 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.612+0000 7fb802d23640 1 librados: setting wanted keys 2024-03-25T19:14:08.617 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.612+0000 7fb802d23640 1 librados: calling monclient init 2024-03-25T19:14:08.617 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.612+0000 7fb802d23640 10 monclient: init 2024-03-25T19:14:08.617 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.613+0000 7fb802d23640 10 monclient: _reopen_session rank -1 2024-03-25T19:14:08.617 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.613+0000 7fb802d23640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:14:08.617 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.613+0000 7fb802d23640 10 monclient(hunting): picked mon.noname-a con 0x7fb7fc0aa210 addr v1:172.21.15.6:6789/0 2024-03-25T19:14:08.617 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.613+0000 7fb802d23640 10 monclient(hunting): picked mon.noname-b con 0x7fb7fc0ab190 addr v1:172.21.15.192:6789/0 2024-03-25T19:14:08.617 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.613+0000 7fb802d23640 10 monclient(hunting): picked mon.noname-c con 0x7fb7fc0ad270 addr v1:172.21.15.6:6790/0 2024-03-25T19:14:08.617 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.613+0000 7fb802d23640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7fc06ea80 con 0x7fb7fc0aa210 2024-03-25T19:14:08.617 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.613+0000 7fb802d23640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7fc1d3880 con 0x7fb7fc0ab190 2024-03-25T19:14:08.618 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.613+0000 7fb802d23640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7fc1d33f0 con 0x7fb7fc0ad270 2024-03-25T19:14:08.618 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.613+0000 7fb802d23640 10 monclient(hunting): _renew_subs 2024-03-25T19:14:08.618 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.613+0000 7fb802d23640 10 monclient(hunting): authenticate will time out at 5970.906516s 2024-03-25T19:14:08.618 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.613+0000 7fb800abf640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb7fc0aa210 0x7fb7fc1d3d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.192:42828/0 (socket says 172.21.15.192:42828) 2024-03-25T19:14:08.618 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.613+0000 7fb800abf640 1 -- 172.21.15.192:0/2029841508 learned_addr learned my addr 172.21.15.192:0/2029841508 (peer_addr_for_me v1:172.21.15.192:0/0) 2024-03-25T19:14:08.618 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.613+0000 7fb7f97fa640 10 client.?.objecter ms_handle_connect 0x7fb7fc0aa210 2024-03-25T19:14:08.618 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.613+0000 7fb7f97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:08.618 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.613+0000 7fb7f97fa640 10 client.?.objecter ms_handle_connect 0x7fb7fc0ab190 2024-03-25T19:14:08.618 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.613+0000 7fb7f97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:08.618 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.614+0000 7fb7f97fa640 10 client.?.objecter ms_handle_connect 0x7fb7fc0ad270 2024-03-25T19:14:08.618 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.614+0000 7fb7f97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:08.619 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.615+0000 7fb7f97fa640 1 -- 172.21.15.192:0/2029841508 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3049952553 0 0) 0x7fb7fc1d3880 con 0x7fb7fc0ab190 2024-03-25T19:14:08.619 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.615+0000 7fb7f97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:14:08.619 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.615+0000 7fb7f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:08.619 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.615+0000 7fb7f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:08.619 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.615+0000 7fb7f97fa640 10 monclient(hunting): my global_id is 4135 2024-03-25T19:14:08.620 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.615+0000 7fb7f97fa640 1 -- 172.21.15.192:0/2029841508 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7e8004eb0 con 0x7fb7fc0ab190 2024-03-25T19:14:08.620 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.615+0000 7fb7f97fa640 1 -- 172.21.15.192:0/2029841508 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3296985475 0 0) 0x7fb7fc1d33f0 con 0x7fb7fc0ad270 2024-03-25T19:14:08.620 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.616+0000 7fb7f97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:14:08.620 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.616+0000 7fb7f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:08.620 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.616+0000 7fb7f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:08.620 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.616+0000 7fb7f97fa640 10 monclient(hunting): my global_id is 4133 2024-03-25T19:14:08.620 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.616+0000 7fb7f97fa640 1 -- 172.21.15.192:0/2029841508 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7fc1d3880 con 0x7fb7fc0ad270 2024-03-25T19:14:08.620 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.616+0000 7fb7f97fa640 1 -- 172.21.15.192:0/2029841508 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2228685057 0 0) 0x7fb7fc06ea80 con 0x7fb7fc0aa210 2024-03-25T19:14:08.620 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.616+0000 7fb7f97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:14:08.620 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.616+0000 7fb7f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:08.620 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.616+0000 7fb7f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:08.620 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.616+0000 7fb7f97fa640 10 monclient(hunting): my global_id is 4137 2024-03-25T19:14:08.620 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.616+0000 7fb7f97fa640 1 -- 172.21.15.192:0/2029841508 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7fc1d33f0 con 0x7fb7fc0aa210 2024-03-25T19:14:08.621 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.620+0000 7fb7f97fa640 1 -- 172.21.15.192:0/2029841508 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1178977335 0 0) 0x7fb7e8004eb0 con 0x7fb7fc0ab190 2024-03-25T19:14:08.622 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.620+0000 7fb7f97fa640 1 -- 172.21.15.192:0/2029841508 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7fc06ea80 con 0x7fb7fc0ab190 2024-03-25T19:14:08.622 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.620+0000 7fb7f97fa640 1 -- 172.21.15.192:0/2029841508 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fb7f0001910 con 0x7fb7fc0ab190 2024-03-25T19:14:08.622 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.620+0000 7fb7f97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:08.622 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.620+0000 7fb7f97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:14:08.622 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.620+0000 7fb7f97fa640 10 monclient(hunting): dump: 2024-03-25T19:14:08.622 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:08.622 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:08.622 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:08.622 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:08.622 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:08.622 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:08.622 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:08.622 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:08.623 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:08.623 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:08.623 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.620+0000 7fb7f97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:14:08.623 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.620+0000 7fb7f97fa640 1 -- 172.21.15.192:0/2029841508 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1913044481 0 0) 0x7fb7fc1d3880 con 0x7fb7fc0ad270 2024-03-25T19:14:08.623 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.620+0000 7fb7f97fa640 1 -- 172.21.15.192:0/2029841508 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7e8004eb0 con 0x7fb7fc0ad270 2024-03-25T19:14:08.623 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.620+0000 7fb7f97fa640 1 -- 172.21.15.192:0/2029841508 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fb7f4001c70 con 0x7fb7fc0ad270 2024-03-25T19:14:08.623 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.620+0000 7fb7f97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:08.623 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.620+0000 7fb7f97fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:14:08.623 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.620+0000 7fb7f97fa640 10 monclient(hunting): dump: 2024-03-25T19:14:08.623 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:08.623 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:08.623 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:08.623 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:08.623 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:08.624 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:08.624 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:08.624 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:08.624 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:08.624 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:08.624 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.621+0000 7fb7f97fa640 1 -- 172.21.15.192:0/2029841508 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 369630789 0 0) 0x7fb7fc1d33f0 con 0x7fb7fc0aa210 2024-03-25T19:14:08.624 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.621+0000 7fb7f97fa640 1 -- 172.21.15.192:0/2029841508 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7fc1d3880 con 0x7fb7fc0aa210 2024-03-25T19:14:08.624 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.621+0000 7fb7f97fa640 1 -- 172.21.15.192:0/2029841508 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fb7ec001b00 con 0x7fb7fc0aa210 2024-03-25T19:14:08.624 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.621+0000 7fb7f97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:08.624 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.621+0000 7fb7f97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:14:08.624 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.621+0000 7fb7f97fa640 10 monclient(hunting): dump: 2024-03-25T19:14:08.624 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:08.624 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:08.624 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:08.624 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:08.625 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:08.625 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:08.625 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:08.625 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:08.625 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:08.625 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:08.626 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.623+0000 7fb7f97fa640 1 -- 172.21.15.192:0/2029841508 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4028787040 0 0) 0x7fb7fc06ea80 con 0x7fb7fc0ab190 2024-03-25T19:14:08.626 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.623+0000 7fb7f97fa640 1 -- 172.21.15.192:0/2029841508 >> v1:172.21.15.6:6790/0 conn(0x7fb7fc0ad270 legacy=0x7fb7fc1dded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:08.626 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.623+0000 7fb7f97fa640 1 -- 172.21.15.192:0/2029841508 >> v1:172.21.15.6:6789/0 conn(0x7fb7fc0aa210 legacy=0x7fb7fc1d3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:08.626 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.623+0000 7fb7f97fa640 10 monclient: _finish_hunting 0 2024-03-25T19:14:08.626 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.623+0000 7fb7f97fa640 1 monclient: found mon.b 2024-03-25T19:14:08.627 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.623+0000 7fb7f97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:14:08.627 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.623+0000 7fb7f97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:08.627 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.623+0000 7fb7f97fa640 1 -- 172.21.15.192:0/2029841508 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7fc1d3580 con 0x7fb7fc0ab190 2024-03-25T19:14:08.627 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.623+0000 7fb7f97fa640 10 monclient: _finish_auth 0 2024-03-25T19:14:08.627 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.623+0000 7fb7f97fa640 10 monclient: _check_auth_tickets 2024-03-25T19:14:08.627 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.623+0000 7fb7f97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:14:08.627 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.623+0000 7fb802d23640 5 monclient: authenticate success, global_id 4135 2024-03-25T19:14:08.627 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.623+0000 7fb802d23640 10 monclient: _renew_subs 2024-03-25T19:14:08.627 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.623+0000 7fb802d23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:08.627 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.623+0000 7fb802d23640 1 -- 172.21.15.192:0/2029841508 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7fc070ff0 con 0x7fb7fc0ab190 2024-03-25T19:14:08.627 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.623+0000 7fb802d23640 10 client.4135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:14:08.627 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.623+0000 7fb802d23640 10 monclient: _renew_subs 2024-03-25T19:14:08.627 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.623+0000 7fb802d23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:08.628 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.623+0000 7fb802d23640 1 -- 172.21.15.192:0/2029841508 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7fc13c8a0 con 0x7fb7fc0ab190 2024-03-25T19:14:08.628 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.623+0000 7fb802d23640 1 librados: init done 2024-03-25T19:14:08.628 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.624+0000 7fb802d23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:08.628 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.624+0000 7fb802d23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:08.628 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.624+0000 7fb802d23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:08.628 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.624+0000 7fb802d23640 1 -- 172.21.15.192:0/2029841508 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7fc0aa800 con 0x7fb7fc0ab190 2024-03-25T19:14:08.652 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-25T19:14:08.652+0000 7fbad1252040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:14:08.720 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.719+0000 7fb7f97fa640 1 -- 172.21.15.192:0/2029841508 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7f00035f0 con 0x7fb7fc0ab190 2024-03-25T19:14:08.720 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.719+0000 7fb7f97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:14:08.720 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.719+0000 7fb7f97fa640 1 -- 172.21.15.192:0/2029841508 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fb7f0003900 con 0x7fb7fc0ab190 2024-03-25T19:14:08.720 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.719+0000 7fb7f97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:08.720 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.719+0000 7fb7f97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:14:08.720 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.719+0000 7fb7f97fa640 10 monclient: dump: 2024-03-25T19:14:08.720 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:08.720 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:08.721 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:08.721 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:08.721 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:08.721 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:08.721 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:08.721 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:08.721 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:08.721 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:08.721 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.719+0000 7fb7f97fa640 1 -- 172.21.15.192:0/2029841508 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb7f0003df0 con 0x7fb7fc0ab190 2024-03-25T19:14:08.721 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.719+0000 7fb7f97fa640 1 -- 172.21.15.192:0/2029841508 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 3544349120 0 0) 0x7fb7f0005440 con 0x7fb7fc0ab190 2024-03-25T19:14:08.721 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.719+0000 7fb7f97fa640 10 client.4135.objecter ms_dispatch 0x7fb7fc000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T19:14:08.721 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.719+0000 7fb7f97fa640 3 client.4135.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T19:14:08.721 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.719+0000 7fb7f97fa640 3 client.4135.objecter handle_osd_map decoding full epoch 5 2024-03-25T19:14:08.721 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.719+0000 7fb7faffd640 4 set_mon_vals no callback set 2024-03-25T19:14:08.721 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.720+0000 7fb7f97fa640 20 client.4135.objecter dump_active .. 0 homeless 2024-03-25T19:14:08.722 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.720+0000 7fb7f97fa640 1 -- 172.21.15.192:0/2029841508 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb7f0005820 con 0x7fb7fc0ab190 2024-03-25T19:14:08.722 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.720+0000 7fb7f97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:08.722 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.720+0000 7fb7f97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:14:08.752 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-25T19:14:08.752+0000 7fbad1252040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T19:14:08.792 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.792+0000 7fb802d23640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d14827e0-9625-4ef5-8adf-d3430bde36e7", "id": 3}] 2024-03-25T19:14:08.792 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.792+0000 7fb802d23640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d14827e0-9625-4ef5-8adf-d3430bde36e7", "id": 3}] 2024-03-25T19:14:08.792 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.792+0000 7fb802d23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:08.792 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.792+0000 7fb802d23640 1 -- 172.21.15.192:0/2029841508 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd new", "uuid": "d14827e0-9625-4ef5-8adf-d3430bde36e7", "id": 3} v 0) v1 -- 0x7fb7fc1d3a10 con 0x7fb7fc0ab190 2024-03-25T19:14:08.795 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-03-25T19:14:08.795+0000 7f1c54213040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T19:14:08.845 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-03-25T19:14:08.845+0000 7f1c54213040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T19:14:08.869 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.868+0000 7fb7f97fa640 1 -- 172.21.15.192:0/2029841508 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d14827e0-9625-4ef5-8adf-d3430bde36e7", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 146962920 0 525046306) 0x7fb7f0004fb0 con 0x7fb7fc0ab190 2024-03-25T19:14:08.869 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.869+0000 7fb7f97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d14827e0-9625-4ef5-8adf-d3430bde36e7", "id": 3}] 2024-03-25T19:14:08.869 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.869+0000 7fb7f97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:14:08.869 INFO:teuthology.orchestra.run.smithi192.stdout:3 2024-03-25T19:14:08.870 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.870+0000 7fb802d23640 10 librados: watch_flush enter 2024-03-25T19:14:08.870 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.870+0000 7fb802d23640 10 librados: watch_flush exit 2024-03-25T19:14:08.871 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.870+0000 7fb802d23640 20 client.4135.objecter shutdown clearing up homeless session... 2024-03-25T19:14:08.871 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.870+0000 7fb802d23640 10 client.4135.objecter successfully canceled tick 2024-03-25T19:14:08.871 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.870+0000 7fb802d23640 10 monclient: shutdown 2024-03-25T19:14:08.871 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.870+0000 7fb802d23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:14:08.871 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.870+0000 7fb802d23640 1 -- 172.21.15.192:0/2029841508 >> v1:172.21.15.192:6789/0 conn(0x7fb7fc0ab190 legacy=0x7fb7fc1d6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:08.871 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.870+0000 7fb802d23640 1 -- 172.21.15.192:0/2029841508 shutdown_connections 2024-03-25T19:14:08.871 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.870+0000 7fb802d23640 1 -- 172.21.15.192:0/2029841508 >> 172.21.15.192:0/2029841508 conn(0x7fb7fc0b3b20 msgr2=0x7fb7fc0b2cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:14:08.871 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.870+0000 7fb802d23640 1 -- 172.21.15.192:0/2029841508 shutdown_connections 2024-03-25T19:14:08.871 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.870+0000 7fb802d23640 1 -- 172.21.15.192:0/2029841508 wait complete. 2024-03-25T19:14:08.871 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.870+0000 7fb802d23640 1 librados: shutdown 2024-03-25T19:14:08.884 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new d1937922-f839-4d21-bd5b-f460adafe633 4 2024-03-25T19:14:08.924 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-25T19:14:08.924+0000 7fbad1252040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:14:08.968 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-25T19:14:08.968+0000 7fbad1252040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T19:14:08.969 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-03-25T19:14:08.969+0000 7f1c54213040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:14:08.983 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.983+0000 7fccf2f82640 10 monclient: get_monmap_and_config 2024-03-25T19:14:08.983 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.983+0000 7fccf2f82640 10 monclient: build_initial_monmap 2024-03-25T19:14:08.983 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.983+0000 7fccf2f82640 10 monclient: monmap: 2024-03-25T19:14:08.983 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2024-03-25T19:14:08.983 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:08.983 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:14:08.984117+0000 2024-03-25T19:14:08.983 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:14:08.984117+0000 2024-03-25T19:14:08.983 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2024-03-25T19:14:08.983 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:08.983 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:14:08.983 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:14:08.983 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:14:08.984 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:08.985 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.983+0000 7fccf2f82640 1 Processor -- start 2024-03-25T19:14:08.985 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.983+0000 7fccf2f82640 1 -- start start 2024-03-25T19:14:08.985 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.983+0000 7fccf2f82640 10 monclient: init 2024-03-25T19:14:08.985 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.984+0000 7fccf2f82640 10 monclient: _reopen_session rank -1 2024-03-25T19:14:08.985 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.984+0000 7fccf2f82640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:14:08.985 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.984+0000 7fccf2f82640 10 monclient(hunting): picked mon.noname-b con 0x7fccec0b4f00 addr v1:172.21.15.6:6790/0 2024-03-25T19:14:08.985 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.984+0000 7fccf2f82640 10 monclient(hunting): picked mon.noname-a con 0x7fccec0b4510 addr v1:172.21.15.6:6789/0 2024-03-25T19:14:08.985 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.984+0000 7fccf2f82640 10 monclient(hunting): picked mon.noname-c con 0x7fccec142ad0 addr v1:172.21.15.192:6789/0 2024-03-25T19:14:08.985 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.984+0000 7fccf2f82640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccec0a9d90 con 0x7fccec0b4510 2024-03-25T19:14:08.986 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.984+0000 7fccf2f82640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccec0aa2b0 con 0x7fccec142ad0 2024-03-25T19:14:08.986 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.984+0000 7fccf2f82640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccec13fc60 con 0x7fccec0b4f00 2024-03-25T19:14:08.986 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.984+0000 7fccf2f82640 10 monclient(hunting): _renew_subs 2024-03-25T19:14:08.986 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.984+0000 7fccf2f82640 10 monclient(hunting): authenticate will time out at 5971.277922s 2024-03-25T19:14:08.986 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.985+0000 7fccebfff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fccec0b4510 0x7fccec13f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.192:42844/0 (socket says 172.21.15.192:42844) 2024-03-25T19:14:08.986 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.985+0000 7fccebfff640 1 -- 172.21.15.192:0/3856794243 learned_addr learned my addr 172.21.15.192:0/3856794243 (peer_addr_for_me v1:172.21.15.192:0/0) 2024-03-25T19:14:08.987 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.987+0000 7fcceb7fe640 1 -- 172.21.15.192:0/3856794243 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1093060065 0 0) 0x7fccec13fc60 con 0x7fccec0b4f00 2024-03-25T19:14:08.987 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.987+0000 7fcceb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:14:08.988 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.987+0000 7fcceb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:08.988 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.987+0000 7fcceb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:08.988 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.987+0000 7fcceb7fe640 10 monclient(hunting): my global_id is 4136 2024-03-25T19:14:08.988 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.987+0000 7fcceb7fe640 1 -- 172.21.15.192:0/3856794243 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccdc003520 con 0x7fccec0b4f00 2024-03-25T19:14:08.988 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.987+0000 7fcceb7fe640 1 -- 172.21.15.192:0/3856794243 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1070276867 0 0) 0x7fccec0aa2b0 con 0x7fccec142ad0 2024-03-25T19:14:08.988 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.987+0000 7fcceb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:14:08.988 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.987+0000 7fcceb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:08.988 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.987+0000 7fcceb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:08.988 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.987+0000 7fcceb7fe640 10 monclient(hunting): my global_id is 4138 2024-03-25T19:14:08.988 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.987+0000 7fcceb7fe640 1 -- 172.21.15.192:0/3856794243 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccec13fc60 con 0x7fccec142ad0 2024-03-25T19:14:08.989 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.989+0000 7fcceb7fe640 1 -- 172.21.15.192:0/3856794243 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2198866268 0 0) 0x7fccec0a9d90 con 0x7fccec0b4510 2024-03-25T19:14:08.989 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.989+0000 7fcceb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:14:08.990 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.989+0000 7fcceb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:08.990 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.989+0000 7fcceb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:08.990 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.989+0000 7fcceb7fe640 10 monclient(hunting): my global_id is 4140 2024-03-25T19:14:08.990 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.989+0000 7fcceb7fe640 1 -- 172.21.15.192:0/3856794243 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccec0aa2b0 con 0x7fccec0b4510 2024-03-25T19:14:08.991 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.990+0000 7fcceb7fe640 1 -- 172.21.15.192:0/3856794243 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3604250523 0 0) 0x7fccec13fc60 con 0x7fccec142ad0 2024-03-25T19:14:08.991 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.990+0000 7fcceb7fe640 1 -- 172.21.15.192:0/3856794243 >> v1:172.21.15.6:6790/0 conn(0x7fccec0b4f00 legacy=0x7fccec1423c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:08.991 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.990+0000 7fcceb7fe640 1 -- 172.21.15.192:0/3856794243 >> v1:172.21.15.6:6789/0 conn(0x7fccec0b4510 legacy=0x7fccec13f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:08.991 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.990+0000 7fcceb7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:14:08.991 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.990+0000 7fcceb7fe640 1 monclient: found mon.noname-c 2024-03-25T19:14:08.991 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.990+0000 7fcceb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:14:08.991 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.990+0000 7fcceb7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:14:08.991 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.990+0000 7fcceb7fe640 1 -- 172.21.15.192:0/3856794243 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccec13fdf0 con 0x7fccec142ad0 2024-03-25T19:14:08.991 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.990+0000 7fcceb7fe640 10 monclient: _finish_auth 0 2024-03-25T19:14:08.991 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.990+0000 7fcceb7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:14:08.992 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.990+0000 7fcceb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:14:08.992 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.990+0000 7fccf2f82640 5 monclient: authenticate success, global_id 4138 2024-03-25T19:14:08.992 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.990+0000 7fccf2f82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:14:08.992 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.990+0000 7fcceb7fe640 1 -- 172.21.15.192:0/3856794243 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fcce40019b0 con 0x7fccec142ad0 2024-03-25T19:14:08.992 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.990+0000 7fcceb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:08.992 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.991+0000 7fcceb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:14:08.992 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.991+0000 7fcceb7fe640 10 monclient: dump: 2024-03-25T19:14:08.992 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:08.992 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:08.992 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:08.992 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:08.992 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:08.992 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:08.992 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:08.992 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:08.993 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:08.993 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:08.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.991+0000 7fccf2f82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:14:08.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.992+0000 7fcceb7fe640 1 -- 172.21.15.192:0/3856794243 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcce4001c00 con 0x7fccec142ad0 2024-03-25T19:14:08.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.992+0000 7fcceb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:14:08.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.992+0000 7fcceb7fe640 1 -- 172.21.15.192:0/3856794243 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fcce4001f90 con 0x7fccec142ad0 2024-03-25T19:14:08.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.992+0000 7fccf2f82640 10 monclient: get_monmap_and_config success 2024-03-25T19:14:08.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.992+0000 7fccf2f82640 4 set_mon_vals no callback set 2024-03-25T19:14:08.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.992+0000 7fcceb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:08.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.992+0000 7fcceb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:14:08.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.992+0000 7fcceb7fe640 10 monclient: dump: 2024-03-25T19:14:08.996 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:08.996 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:08.996 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:08.997 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:08.997 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:08.997 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:08.997 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:08.997 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:08.997 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:08.997 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:08.997 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.993+0000 7fccf2f82640 10 monclient: shutdown 2024-03-25T19:14:08.997 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.993+0000 7fccf2f82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:14:08.997 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.993+0000 7fccf2f82640 1 -- 172.21.15.192:0/3856794243 >> v1:172.21.15.192:6789/0 conn(0x7fccec142ad0 legacy=0x7fccec14dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:08.997 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.993+0000 7fccf2f82640 1 -- 172.21.15.192:0/3856794243 shutdown_connections 2024-03-25T19:14:08.997 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.993+0000 7fccf2f82640 1 -- 172.21.15.192:0/3856794243 >> 172.21.15.192:0/3856794243 conn(0x7fccec066b00 msgr2=0x7fccec066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:14:08.997 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.993+0000 7fccf2f82640 1 -- 172.21.15.192:0/3856794243 shutdown_connections 2024-03-25T19:14:08.997 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.993+0000 7fccf2f82640 1 -- 172.21.15.192:0/3856794243 wait complete. 2024-03-25T19:14:08.997 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.994+0000 7fccf2f82640 10 monclient: build_initial_monmap 2024-03-25T19:14:08.998 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.994+0000 7fccf2f82640 10 monclient: monmap: 2024-03-25T19:14:08.998 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2024-03-25T19:14:08.998 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:14:08.998 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 0.000000 2024-03-25T19:14:08.998 INFO:teuthology.orchestra.run.smithi192.stderr:created 0.000000 2024-03-25T19:14:08.998 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2024-03-25T19:14:08.998 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2024-03-25T19:14:08.998 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:14:08.998 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:14:08.998 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:14:08.998 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:08.998 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.994+0000 7fccf2f82640 1 librados: starting msgr at 2024-03-25T19:14:08.998 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.994+0000 7fccf2f82640 1 librados: starting objecter 2024-03-25T19:14:08.998 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.994+0000 7fccf2f82640 1 Processor -- start 2024-03-25T19:14:08.999 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.994+0000 7fccf2f82640 1 -- start start 2024-03-25T19:14:08.999 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.994+0000 7fccf2f82640 1 librados: setting wanted keys 2024-03-25T19:14:08.999 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.994+0000 7fccf2f82640 1 librados: calling monclient init 2024-03-25T19:14:08.999 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.994+0000 7fccf2f82640 10 monclient: init 2024-03-25T19:14:08.999 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.995+0000 7fccf2f82640 10 monclient: _reopen_session rank -1 2024-03-25T19:14:08.999 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.995+0000 7fccf2f82640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:14:08.999 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.995+0000 7fccf2f82640 10 monclient(hunting): picked mon.noname-c con 0x7fccec0b4510 addr v1:172.21.15.6:6790/0 2024-03-25T19:14:08.999 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.995+0000 7fccf2f82640 10 monclient(hunting): picked mon.noname-b con 0x7fccec0b4f00 addr v1:172.21.15.192:6789/0 2024-03-25T19:14:08.999 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.995+0000 7fccf2f82640 10 monclient(hunting): picked mon.noname-a con 0x7fccec142ad0 addr v1:172.21.15.6:6789/0 2024-03-25T19:14:08.999 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.995+0000 7fccf2f82640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccec0ac330 con 0x7fccec142ad0 2024-03-25T19:14:08.999 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.995+0000 7fccf2f82640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccec0b8350 con 0x7fccec0b4f00 2024-03-25T19:14:08.999 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.995+0000 7fccf2f82640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccec145bd0 con 0x7fccec0b4510 2024-03-25T19:14:08.999 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.995+0000 7fccf2f82640 10 monclient(hunting): _renew_subs 2024-03-25T19:14:08.999 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.995+0000 7fccf2f82640 10 monclient(hunting): authenticate will time out at 5971.288814s 2024-03-25T19:14:08.999 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.995+0000 7fccf151f640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fccec142ad0 0x7fccec1d7230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.192:42846/0 (socket says 172.21.15.192:42846) 2024-03-25T19:14:09.000 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.995+0000 7fccf151f640 1 -- 172.21.15.192:0/2385325262 learned_addr learned my addr 172.21.15.192:0/2385325262 (peer_addr_for_me v1:172.21.15.192:0/0) 2024-03-25T19:14:09.000 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.996+0000 7fcce97fa640 10 client.?.objecter ms_handle_connect 0x7fccec142ad0 2024-03-25T19:14:09.000 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.996+0000 7fcce97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:09.000 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.996+0000 7fcce97fa640 10 client.?.objecter ms_handle_connect 0x7fccec0b4f00 2024-03-25T19:14:09.000 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.996+0000 7fcce97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:09.000 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.996+0000 7fcce97fa640 10 client.?.objecter ms_handle_connect 0x7fccec0b4510 2024-03-25T19:14:09.000 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.996+0000 7fcce97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:09.001 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.998+0000 7fcce97fa640 1 -- 172.21.15.192:0/2385325262 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3422065383 0 0) 0x7fccec0b8350 con 0x7fccec0b4f00 2024-03-25T19:14:09.001 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.998+0000 7fcce97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:14:09.001 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.998+0000 7fcce97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:09.001 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.998+0000 7fcce97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:09.001 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.998+0000 7fcce97fa640 10 monclient(hunting): my global_id is 4141 2024-03-25T19:14:09.001 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.998+0000 7fcce97fa640 1 -- 172.21.15.192:0/2385325262 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccd0004eb0 con 0x7fccec0b4f00 2024-03-25T19:14:09.001 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.998+0000 7fcce97fa640 1 -- 172.21.15.192:0/2385325262 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3366970141 0 0) 0x7fccec0ac330 con 0x7fccec142ad0 2024-03-25T19:14:09.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.998+0000 7fcce97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:14:09.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.998+0000 7fcce97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:09.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.998+0000 7fcce97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:09.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.998+0000 7fcce97fa640 10 monclient(hunting): my global_id is 4143 2024-03-25T19:14:09.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.998+0000 7fcce97fa640 1 -- 172.21.15.192:0/2385325262 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccec0b8350 con 0x7fccec142ad0 2024-03-25T19:14:09.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.998+0000 7fcce97fa640 1 -- 172.21.15.192:0/2385325262 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 76649881 0 0) 0x7fccec145bd0 con 0x7fccec0b4510 2024-03-25T19:14:09.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.998+0000 7fcce97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:14:09.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.998+0000 7fcce97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:09.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.998+0000 7fcce97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:09.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.998+0000 7fcce97fa640 10 monclient(hunting): my global_id is 4139 2024-03-25T19:14:09.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:08.998+0000 7fcce97fa640 1 -- 172.21.15.192:0/2385325262 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccec0ac330 con 0x7fccec0b4510 2024-03-25T19:14:09.003 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.002+0000 7fcce97fa640 1 -- 172.21.15.192:0/2385325262 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2235259906 0 0) 0x7fccd0004eb0 con 0x7fccec0b4f00 2024-03-25T19:14:09.003 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.002+0000 7fcce97fa640 1 -- 172.21.15.192:0/2385325262 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fccec145bd0 con 0x7fccec0b4f00 2024-03-25T19:14:09.003 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.002+0000 7fcce97fa640 1 -- 172.21.15.192:0/2385325262 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fccd80018f0 con 0x7fccec0b4f00 2024-03-25T19:14:09.003 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.002+0000 7fcce97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:09.003 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.002+0000 7fcce97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:14:09.003 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.002+0000 7fcce97fa640 10 monclient(hunting): dump: 2024-03-25T19:14:09.003 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:09.003 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:09.003 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:09.003 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:09.004 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:09.004 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:09.004 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:09.004 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:09.004 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:09.004 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:09.004 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.002+0000 7fcce97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:14:09.004 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.003+0000 7fcce97fa640 1 -- 172.21.15.192:0/2385325262 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3670558567 0 0) 0x7fccec0ac330 con 0x7fccec0b4510 2024-03-25T19:14:09.005 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.003+0000 7fcce97fa640 1 -- 172.21.15.192:0/2385325262 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fccd0004eb0 con 0x7fccec0b4510 2024-03-25T19:14:09.005 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.003+0000 7fcce97fa640 1 -- 172.21.15.192:0/2385325262 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fcce0001710 con 0x7fccec0b4510 2024-03-25T19:14:09.005 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.003+0000 7fcce97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:09.005 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.003+0000 7fcce97fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:14:09.005 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.003+0000 7fcce97fa640 10 monclient(hunting): dump: 2024-03-25T19:14:09.005 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:09.005 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:09.005 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:09.005 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:09.005 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:09.005 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:09.005 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:09.006 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:09.006 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:09.006 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:09.006 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.004+0000 7fcce97fa640 1 -- 172.21.15.192:0/2385325262 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4207340505 0 0) 0x7fccec0b8350 con 0x7fccec142ad0 2024-03-25T19:14:09.006 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.004+0000 7fcce97fa640 1 -- 172.21.15.192:0/2385325262 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fccec0ac330 con 0x7fccec142ad0 2024-03-25T19:14:09.006 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.004+0000 7fcce97fa640 1 -- 172.21.15.192:0/2385325262 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fcce4001f10 con 0x7fccec142ad0 2024-03-25T19:14:09.006 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.004+0000 7fcce97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:09.006 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.004+0000 7fcce97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:14:09.006 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.004+0000 7fcce97fa640 10 monclient(hunting): dump: 2024-03-25T19:14:09.006 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:09.006 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:09.006 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:09.006 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:09.006 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:09.007 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:09.007 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:09.007 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:09.007 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:09.007 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:09.008 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.006+0000 7fcce97fa640 1 -- 172.21.15.192:0/2385325262 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2869183229 0 0) 0x7fccec145bd0 con 0x7fccec0b4f00 2024-03-25T19:14:09.008 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.006+0000 7fcce97fa640 1 -- 172.21.15.192:0/2385325262 >> v1:172.21.15.6:6790/0 conn(0x7fccec0b4510 legacy=0x7fccec0b7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:09.008 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.006+0000 7fcce97fa640 1 -- 172.21.15.192:0/2385325262 >> v1:172.21.15.6:6789/0 conn(0x7fccec142ad0 legacy=0x7fccec1d7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:09.008 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.006+0000 7fcce97fa640 10 monclient: _finish_hunting 0 2024-03-25T19:14:09.008 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.006+0000 7fcce97fa640 1 monclient: found mon.b 2024-03-25T19:14:09.008 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.006+0000 7fcce97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:14:09.008 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.006+0000 7fcce97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:09.009 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.006+0000 7fcce97fa640 1 -- 172.21.15.192:0/2385325262 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccec145d60 con 0x7fccec0b4f00 2024-03-25T19:14:09.009 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.006+0000 7fcce97fa640 10 monclient: _finish_auth 0 2024-03-25T19:14:09.009 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.006+0000 7fcce97fa640 10 monclient: _check_auth_tickets 2024-03-25T19:14:09.009 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.006+0000 7fcce97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:14:09.009 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.006+0000 7fccf2f82640 5 monclient: authenticate success, global_id 4141 2024-03-25T19:14:09.009 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.006+0000 7fccf2f82640 10 monclient: _renew_subs 2024-03-25T19:14:09.009 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.006+0000 7fccf2f82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:09.009 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.006+0000 7fccf2f82640 1 -- 172.21.15.192:0/2385325262 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccec0b7530 con 0x7fccec0b4f00 2024-03-25T19:14:09.009 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.006+0000 7fccf2f82640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:14:09.009 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.006+0000 7fccf2f82640 10 monclient: _renew_subs 2024-03-25T19:14:09.009 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.006+0000 7fccf2f82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:09.009 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.006+0000 7fccf2f82640 1 -- 172.21.15.192:0/2385325262 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fccec137890 con 0x7fccec0b4f00 2024-03-25T19:14:09.010 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.006+0000 7fccf2f82640 1 librados: init done 2024-03-25T19:14:09.010 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.007+0000 7fccf2f82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:09.010 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.007+0000 7fccf2f82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:09.010 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.007+0000 7fccf2f82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:09.010 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.007+0000 7fccf2f82640 1 -- 172.21.15.192:0/2385325262 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccec04f730 con 0x7fccec0b4f00 2024-03-25T19:14:09.069 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-03-25T19:14:09.069+0000 7f1c54213040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T19:14:09.107 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.107+0000 7fcce97fa640 1 -- 172.21.15.192:0/2385325262 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fccd8001d40 con 0x7fccec0b4f00 2024-03-25T19:14:09.108 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.107+0000 7fcce97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:14:09.108 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.107+0000 7fcce97fa640 1 -- 172.21.15.192:0/2385325262 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fccd8004a00 con 0x7fccec0b4f00 2024-03-25T19:14:09.108 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.107+0000 7fcce97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:09.108 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.107+0000 7fcce97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:14:09.108 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.107+0000 7fcce97fa640 10 monclient: dump: 2024-03-25T19:14:09.108 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:09.108 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:09.108 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:09.108 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:09.108 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:09.108 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:09.108 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:09.109 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:09.109 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:09.109 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:09.109 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.107+0000 7fcce97fa640 1 -- 172.21.15.192:0/2385325262 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fccd8004fb0 con 0x7fccec0b4f00 2024-03-25T19:14:09.109 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.107+0000 7fcceaffd640 4 set_mon_vals no callback set 2024-03-25T19:14:09.109 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.107+0000 7fcce97fa640 1 -- 172.21.15.192:0/2385325262 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 4114812050 0 0) 0x7fccd8006680 con 0x7fccec0b4f00 2024-03-25T19:14:09.109 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.107+0000 7fcce97fa640 10 client.4141.objecter ms_dispatch 0x7fccec000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T19:14:09.109 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.107+0000 7fcce97fa640 3 client.4141.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T19:14:09.109 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.107+0000 7fcce97fa640 3 client.4141.objecter handle_osd_map decoding full epoch 6 2024-03-25T19:14:09.109 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.107+0000 7fcce97fa640 20 client.4141.objecter dump_active .. 0 homeless 2024-03-25T19:14:09.109 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.107+0000 7fcce97fa640 1 -- 172.21.15.192:0/2385325262 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fccd8006a60 con 0x7fccec0b4f00 2024-03-25T19:14:09.109 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.107+0000 7fcce97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:09.110 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.107+0000 7fcce97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:14:09.138 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-25T19:14:09.138+0000 7fbad1252040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T19:14:09.181 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.181+0000 7fccf2f82640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d1937922-f839-4d21-bd5b-f460adafe633", "id": 4}] 2024-03-25T19:14:09.181 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.181+0000 7fccf2f82640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d1937922-f839-4d21-bd5b-f460adafe633", "id": 4}] 2024-03-25T19:14:09.181 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.181+0000 7fccf2f82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:09.181 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.181+0000 7fccf2f82640 1 -- 172.21.15.192:0/2385325262 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd new", "uuid": "d1937922-f839-4d21-bd5b-f460adafe633", "id": 4} v 0) v1 -- 0x7fccec13e7f0 con 0x7fccec0b4f00 2024-03-25T19:14:09.241 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-03-25T19:14:09.241+0000 7f1c54213040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:14:09.257 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.257+0000 7fcce97fa640 1 -- 172.21.15.192:0/2385325262 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d1937922-f839-4d21-bd5b-f460adafe633", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 3988588834 0 1697012583) 0x7fccd8006240 con 0x7fccec0b4f00 2024-03-25T19:14:09.257 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.257+0000 7fcce97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d1937922-f839-4d21-bd5b-f460adafe633", "id": 4}] 2024-03-25T19:14:09.258 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.257+0000 7fcce97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:14:09.258 INFO:teuthology.orchestra.run.smithi192.stdout:4 2024-03-25T19:14:09.259 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.258+0000 7fccf2f82640 10 librados: watch_flush enter 2024-03-25T19:14:09.259 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.258+0000 7fccf2f82640 10 librados: watch_flush exit 2024-03-25T19:14:09.259 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.258+0000 7fccf2f82640 20 client.4141.objecter shutdown clearing up homeless session... 2024-03-25T19:14:09.259 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.258+0000 7fccf2f82640 10 client.4141.objecter successfully canceled tick 2024-03-25T19:14:09.259 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.258+0000 7fccf2f82640 10 monclient: shutdown 2024-03-25T19:14:09.259 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.258+0000 7fccf2f82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:14:09.259 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.258+0000 7fccf2f82640 1 -- 172.21.15.192:0/2385325262 >> v1:172.21.15.192:6789/0 conn(0x7fccec0b4f00 legacy=0x7fccec137180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:09.259 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.258+0000 7fccf2f82640 1 -- 172.21.15.192:0/2385325262 shutdown_connections 2024-03-25T19:14:09.260 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.258+0000 7fccf2f82640 1 -- 172.21.15.192:0/2385325262 >> 172.21.15.192:0/2385325262 conn(0x7fccec066b00 msgr2=0x7fccec13b830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:14:09.260 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.259+0000 7fccf2f82640 1 -- 172.21.15.192:0/2385325262 shutdown_connections 2024-03-25T19:14:09.260 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.259+0000 7fccf2f82640 1 -- 172.21.15.192:0/2385325262 wait complete. 2024-03-25T19:14:09.260 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.259+0000 7fccf2f82640 1 librados: shutdown 2024-03-25T19:14:09.269 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new 31abd8d9-93e5-4f4b-928e-d044c4b46999 5 2024-03-25T19:14:09.287 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-03-25T19:14:09.287+0000 7f1c54213040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T19:14:09.368 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.368+0000 7f88d59cf640 10 monclient: get_monmap_and_config 2024-03-25T19:14:09.368 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.368+0000 7f88d59cf640 10 monclient: build_initial_monmap 2024-03-25T19:14:09.368 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.368+0000 7f88d59cf640 10 monclient: monmap: 2024-03-25T19:14:09.368 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2024-03-25T19:14:09.369 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:09.369 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:14:09.369654+0000 2024-03-25T19:14:09.369 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:14:09.369654+0000 2024-03-25T19:14:09.369 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2024-03-25T19:14:09.369 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:09.369 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:14:09.369 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:14:09.369 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:14:09.369 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:09.371 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.369+0000 7f88d59cf640 1 Processor -- start 2024-03-25T19:14:09.371 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.369+0000 7f88d59cf640 1 -- start start 2024-03-25T19:14:09.371 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.369+0000 7f88d59cf640 10 monclient: init 2024-03-25T19:14:09.371 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.369+0000 7f88d59cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:14:09.371 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.369+0000 7f88d59cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:14:09.371 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.369+0000 7f88d59cf640 10 monclient(hunting): picked mon.noname-c con 0x7f88d006c360 addr v1:172.21.15.192:6789/0 2024-03-25T19:14:09.371 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.369+0000 7f88d59cf640 10 monclient(hunting): picked mon.noname-a con 0x7f88d006c740 addr v1:172.21.15.6:6789/0 2024-03-25T19:14:09.371 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.369+0000 7f88d59cf640 10 monclient(hunting): picked mon.noname-b con 0x7f88d0146060 addr v1:172.21.15.6:6790/0 2024-03-25T19:14:09.371 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.369+0000 7f88d59cf640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88d0140040 con 0x7f88d006c740 2024-03-25T19:14:09.371 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.369+0000 7f88d59cf640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88d0070650 con 0x7f88d006c360 2024-03-25T19:14:09.371 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.369+0000 7f88d59cf640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88d00708c0 con 0x7f88d0146060 2024-03-25T19:14:09.371 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.369+0000 7f88d59cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:14:09.371 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.369+0000 7f88d59cf640 10 monclient(hunting): authenticate will time out at 5971.663262s 2024-03-25T19:14:09.371 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.370+0000 7f88ce7fc640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f88d006c740 0x7f88d0145950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.192:42860/0 (socket says 172.21.15.192:42860) 2024-03-25T19:14:09.372 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.370+0000 7f88ce7fc640 1 -- 172.21.15.192:0/2945000291 learned_addr learned my addr 172.21.15.192:0/2945000291 (peer_addr_for_me v1:172.21.15.192:0/0) 2024-03-25T19:14:09.373 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.372+0000 7f88cdffb640 1 -- 172.21.15.192:0/2945000291 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 962064605 0 0) 0x7f88d0140040 con 0x7f88d006c740 2024-03-25T19:14:09.373 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.372+0000 7f88cdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:14:09.373 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.372+0000 7f88cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:09.373 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.372+0000 7f88cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:09.373 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.372+0000 7f88cdffb640 10 monclient(hunting): my global_id is 4146 2024-03-25T19:14:09.373 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.372+0000 7f88cdffb640 1 -- 172.21.15.192:0/2945000291 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88b8003520 con 0x7f88d006c740 2024-03-25T19:14:09.373 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.372+0000 7f88cdffb640 1 -- 172.21.15.192:0/2945000291 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 698652147 0 0) 0x7f88d0070650 con 0x7f88d006c360 2024-03-25T19:14:09.373 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.372+0000 7f88cdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:14:09.373 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.372+0000 7f88cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:09.373 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.372+0000 7f88cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:09.373 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.372+0000 7f88cdffb640 10 monclient(hunting): my global_id is 4144 2024-03-25T19:14:09.374 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.372+0000 7f88cdffb640 1 -- 172.21.15.192:0/2945000291 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88d0140040 con 0x7f88d006c360 2024-03-25T19:14:09.374 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.373+0000 7f88cdffb640 1 -- 172.21.15.192:0/2945000291 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2885076070 0 0) 0x7f88d00708c0 con 0x7f88d0146060 2024-03-25T19:14:09.374 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.373+0000 7f88cdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:14:09.374 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.373+0000 7f88cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:09.374 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.373+0000 7f88cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:09.374 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.373+0000 7f88cdffb640 10 monclient(hunting): my global_id is 4142 2024-03-25T19:14:09.374 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.373+0000 7f88cdffb640 1 -- 172.21.15.192:0/2945000291 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88d0070650 con 0x7f88d0146060 2024-03-25T19:14:09.376 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.375+0000 7f88cdffb640 1 -- 172.21.15.192:0/2945000291 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1597622009 0 0) 0x7f88d0140040 con 0x7f88d006c360 2024-03-25T19:14:09.376 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.375+0000 7f88cdffb640 1 -- 172.21.15.192:0/2945000291 >> v1:172.21.15.6:6790/0 conn(0x7f88d0146060 legacy=0x7f88d014dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:09.376 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.375+0000 7f88cdffb640 1 -- 172.21.15.192:0/2945000291 >> v1:172.21.15.6:6789/0 conn(0x7f88d006c740 legacy=0x7f88d0145950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:09.376 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.375+0000 7f88cdffb640 10 monclient: _finish_hunting 0 2024-03-25T19:14:09.376 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.375+0000 7f88cdffb640 1 monclient: found mon.noname-c 2024-03-25T19:14:09.376 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.375+0000 7f88cdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:14:09.376 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.375+0000 7f88cdffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:14:09.376 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.375+0000 7f88cdffb640 1 -- 172.21.15.192:0/2945000291 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88d006d210 con 0x7f88d006c360 2024-03-25T19:14:09.376 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.375+0000 7f88cdffb640 10 monclient: _finish_auth 0 2024-03-25T19:14:09.376 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.375+0000 7f88cdffb640 10 monclient: _check_auth_tickets 2024-03-25T19:14:09.376 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.375+0000 7f88cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:14:09.376 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.375+0000 7f88d59cf640 5 monclient: authenticate success, global_id 4144 2024-03-25T19:14:09.377 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.375+0000 7f88d59cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:14:09.377 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.375+0000 7f88cdffb640 1 -- 172.21.15.192:0/2945000291 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f88c0001630 con 0x7f88d006c360 2024-03-25T19:14:09.377 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.375+0000 7f88cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:09.377 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.376+0000 7f88cdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:14:09.377 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.376+0000 7f88cdffb640 10 monclient: dump: 2024-03-25T19:14:09.377 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:09.377 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:09.377 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:09.377 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:09.377 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:09.377 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:09.377 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:09.377 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:09.377 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:09.377 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:09.377 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.376+0000 7f88d59cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:14:09.379 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.377+0000 7f88cdffb640 1 -- 172.21.15.192:0/2945000291 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88c0001880 con 0x7f88d006c360 2024-03-25T19:14:09.379 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.377+0000 7f88cdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:14:09.379 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.377+0000 7f88cdffb640 1 -- 172.21.15.192:0/2945000291 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f88c0001c10 con 0x7f88d006c360 2024-03-25T19:14:09.379 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.377+0000 7f88d59cf640 10 monclient: get_monmap_and_config success 2024-03-25T19:14:09.379 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.377+0000 7f88d59cf640 4 set_mon_vals no callback set 2024-03-25T19:14:09.380 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.377+0000 7f88cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:09.380 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.377+0000 7f88cdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:14:09.380 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.377+0000 7f88cdffb640 10 monclient: dump: 2024-03-25T19:14:09.380 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:09.380 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:09.380 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:09.380 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:09.380 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:09.380 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:09.380 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:09.380 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:09.380 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:09.380 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:09.380 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.378+0000 7f88d59cf640 10 monclient: shutdown 2024-03-25T19:14:09.380 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.378+0000 7f88d59cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:14:09.381 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.378+0000 7f88d59cf640 1 -- 172.21.15.192:0/2945000291 >> v1:172.21.15.192:6789/0 conn(0x7f88d006c360 legacy=0x7f88d00723e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:09.381 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.378+0000 7f88d59cf640 1 -- 172.21.15.192:0/2945000291 shutdown_connections 2024-03-25T19:14:09.381 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.378+0000 7f88d59cf640 1 -- 172.21.15.192:0/2945000291 >> 172.21.15.192:0/2945000291 conn(0x7f88d00b5ce0 msgr2=0x7f88d00b60c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:14:09.381 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.378+0000 7f88d59cf640 1 -- 172.21.15.192:0/2945000291 shutdown_connections 2024-03-25T19:14:09.381 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.378+0000 7f88d59cf640 1 -- 172.21.15.192:0/2945000291 wait complete. 2024-03-25T19:14:09.381 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.379+0000 7f88d59cf640 10 monclient: build_initial_monmap 2024-03-25T19:14:09.381 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.379+0000 7f88d59cf640 10 monclient: monmap: 2024-03-25T19:14:09.381 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2024-03-25T19:14:09.381 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:14:09.381 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 0.000000 2024-03-25T19:14:09.381 INFO:teuthology.orchestra.run.smithi192.stderr:created 0.000000 2024-03-25T19:14:09.381 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2024-03-25T19:14:09.381 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2024-03-25T19:14:09.381 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:14:09.381 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:14:09.382 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:14:09.382 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:09.382 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.379+0000 7f88d59cf640 1 librados: starting msgr at 2024-03-25T19:14:09.382 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.379+0000 7f88d59cf640 1 librados: starting objecter 2024-03-25T19:14:09.382 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.379+0000 7f88d59cf640 1 Processor -- start 2024-03-25T19:14:09.382 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.379+0000 7f88d59cf640 1 -- start start 2024-03-25T19:14:09.382 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.379+0000 7f88d59cf640 1 librados: setting wanted keys 2024-03-25T19:14:09.382 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.379+0000 7f88d59cf640 1 librados: calling monclient init 2024-03-25T19:14:09.382 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.379+0000 7f88d59cf640 10 monclient: init 2024-03-25T19:14:09.384 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.380+0000 7f88d59cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:14:09.384 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.380+0000 7f88d59cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:14:09.384 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.380+0000 7f88d59cf640 10 monclient(hunting): picked mon.noname-b con 0x7f88d006c360 addr v1:172.21.15.192:6789/0 2024-03-25T19:14:09.384 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.380+0000 7f88d59cf640 10 monclient(hunting): picked mon.noname-a con 0x7f88d006c740 addr v1:172.21.15.6:6789/0 2024-03-25T19:14:09.384 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.380+0000 7f88d59cf640 10 monclient(hunting): picked mon.noname-c con 0x7f88d0146060 addr v1:172.21.15.6:6790/0 2024-03-25T19:14:09.385 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.380+0000 7f88d59cf640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88d00aa4f0 con 0x7f88d006c740 2024-03-25T19:14:09.385 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.380+0000 7f88d59cf640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88d00ac2c0 con 0x7f88d006c360 2024-03-25T19:14:09.385 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.380+0000 7f88d59cf640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88d00ae390 con 0x7f88d0146060 2024-03-25T19:14:09.385 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.380+0000 7f88d59cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:14:09.385 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.380+0000 7f88d59cf640 10 monclient(hunting): authenticate will time out at 5971.673766s 2024-03-25T19:14:09.385 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.380+0000 7f88ceffd640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f88d006c360 0x7f88d00aa980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.192:58420/0 (socket says 172.21.15.192:58420) 2024-03-25T19:14:09.385 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.380+0000 7f88ceffd640 1 -- 172.21.15.192:0/3031048156 learned_addr learned my addr 172.21.15.192:0/3031048156 (peer_addr_for_me v1:172.21.15.192:0/0) 2024-03-25T19:14:09.385 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.381+0000 7f88b7fff640 10 client.?.objecter ms_handle_connect 0x7f88d006c360 2024-03-25T19:14:09.385 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.381+0000 7f88b7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:09.385 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.381+0000 7f88b7fff640 10 client.?.objecter ms_handle_connect 0x7f88d006c740 2024-03-25T19:14:09.385 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.381+0000 7f88b7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:09.385 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.381+0000 7f88b7fff640 10 client.?.objecter ms_handle_connect 0x7f88d0146060 2024-03-25T19:14:09.385 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.381+0000 7f88b7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:09.385 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.382+0000 7f88b7fff640 1 -- 172.21.15.192:0/3031048156 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1256032323 0 0) 0x7f88d00aa4f0 con 0x7f88d006c740 2024-03-25T19:14:09.386 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.382+0000 7f88b7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:14:09.386 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.382+0000 7f88b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:09.386 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.382+0000 7f88b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:09.386 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.382+0000 7f88b7fff640 10 monclient(hunting): my global_id is 4149 2024-03-25T19:14:09.386 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.382+0000 7f88b7fff640 1 -- 172.21.15.192:0/3031048156 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88bc004eb0 con 0x7f88d006c740 2024-03-25T19:14:09.386 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.383+0000 7f88b7fff640 1 -- 172.21.15.192:0/3031048156 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2886070255 0 0) 0x7f88d00ac2c0 con 0x7f88d006c360 2024-03-25T19:14:09.386 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.383+0000 7f88b7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:14:09.386 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.383+0000 7f88b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:09.386 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.383+0000 7f88b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:09.386 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.383+0000 7f88b7fff640 10 monclient(hunting): my global_id is 4147 2024-03-25T19:14:09.386 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.383+0000 7f88b7fff640 1 -- 172.21.15.192:0/3031048156 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88d00aa4f0 con 0x7f88d006c360 2024-03-25T19:14:09.386 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.383+0000 7f88b7fff640 1 -- 172.21.15.192:0/3031048156 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2529337044 0 0) 0x7f88d00ae390 con 0x7f88d0146060 2024-03-25T19:14:09.386 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.383+0000 7f88b7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:14:09.387 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.383+0000 7f88b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:09.387 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.383+0000 7f88b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:09.387 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.383+0000 7f88b7fff640 10 monclient(hunting): my global_id is 4145 2024-03-25T19:14:09.387 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.383+0000 7f88b7fff640 1 -- 172.21.15.192:0/3031048156 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88d00ac2c0 con 0x7f88d0146060 2024-03-25T19:14:09.388 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.387+0000 7f88b7fff640 1 -- 172.21.15.192:0/3031048156 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2956431642 0 0) 0x7f88bc004eb0 con 0x7f88d006c740 2024-03-25T19:14:09.388 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.387+0000 7f88b7fff640 1 -- 172.21.15.192:0/3031048156 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88d00ae390 con 0x7f88d006c740 2024-03-25T19:14:09.388 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.387+0000 7f88b7fff640 1 -- 172.21.15.192:0/3031048156 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f88c4001950 con 0x7f88d006c740 2024-03-25T19:14:09.388 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.387+0000 7f88b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:09.388 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.387+0000 7f88b7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:14:09.388 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.387+0000 7f88b7fff640 10 monclient(hunting): dump: 2024-03-25T19:14:09.388 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:09.389 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:09.389 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:09.389 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:09.389 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:09.389 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:09.389 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:09.389 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:09.389 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:09.389 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:09.389 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.387+0000 7f88b7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:14:09.389 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.387+0000 7f88b7fff640 1 -- 172.21.15.192:0/3031048156 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3683665014 0 0) 0x7f88d00aa4f0 con 0x7f88d006c360 2024-03-25T19:14:09.389 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.387+0000 7f88b7fff640 1 -- 172.21.15.192:0/3031048156 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88bc004eb0 con 0x7f88d006c360 2024-03-25T19:14:09.389 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.387+0000 7f88b7fff640 1 -- 172.21.15.192:0/3031048156 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f88c0001c10 con 0x7f88d006c360 2024-03-25T19:14:09.389 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.387+0000 7f88b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:09.390 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.387+0000 7f88b7fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:14:09.390 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.387+0000 7f88b7fff640 10 monclient(hunting): dump: 2024-03-25T19:14:09.390 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:09.390 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:09.390 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:09.390 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:09.390 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:09.390 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:09.390 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:09.390 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:09.390 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:09.390 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:09.390 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.388+0000 7f88b7fff640 1 -- 172.21.15.192:0/3031048156 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3910253939 0 0) 0x7f88d00ac2c0 con 0x7f88d0146060 2024-03-25T19:14:09.390 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.388+0000 7f88b7fff640 1 -- 172.21.15.192:0/3031048156 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88d00aa4f0 con 0x7f88d0146060 2024-03-25T19:14:09.391 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.388+0000 7f88b7fff640 1 -- 172.21.15.192:0/3031048156 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f88c8001c70 con 0x7f88d0146060 2024-03-25T19:14:09.391 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.388+0000 7f88b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:09.391 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.388+0000 7f88b7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:14:09.391 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.388+0000 7f88b7fff640 10 monclient(hunting): dump: 2024-03-25T19:14:09.391 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:09.391 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:09.391 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:09.391 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:09.391 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:09.391 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:09.391 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:09.391 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:09.391 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:09.391 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:09.393 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.391+0000 7f88b7fff640 1 -- 172.21.15.192:0/3031048156 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3642203527 0 0) 0x7f88bc004eb0 con 0x7f88d006c360 2024-03-25T19:14:09.393 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.391+0000 7f88b7fff640 1 -- 172.21.15.192:0/3031048156 >> v1:172.21.15.6:6790/0 conn(0x7f88d0146060 legacy=0x7f88d01e2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:09.393 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.391+0000 7f88b7fff640 1 -- 172.21.15.192:0/3031048156 >> v1:172.21.15.6:6789/0 conn(0x7f88d006c740 legacy=0x7f88d0142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:09.393 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.391+0000 7f88b7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:14:09.393 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.391+0000 7f88b7fff640 1 monclient: found mon.b 2024-03-25T19:14:09.393 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.391+0000 7f88b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:14:09.393 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.391+0000 7f88b7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:09.393 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.391+0000 7f88b7fff640 1 -- 172.21.15.192:0/3031048156 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88d01db6e0 con 0x7f88d006c360 2024-03-25T19:14:09.393 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.391+0000 7f88b7fff640 10 monclient: _finish_auth 0 2024-03-25T19:14:09.393 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.391+0000 7f88b7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:14:09.393 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.391+0000 7f88b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:14:09.393 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.391+0000 7f88d59cf640 5 monclient: authenticate success, global_id 4147 2024-03-25T19:14:09.393 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.391+0000 7f88d59cf640 10 monclient: _renew_subs 2024-03-25T19:14:09.393 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.391+0000 7f88d59cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:09.394 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.391+0000 7f88d59cf640 1 -- 172.21.15.192:0/3031048156 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88d01db930 con 0x7f88d006c360 2024-03-25T19:14:09.394 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.391+0000 7f88d59cf640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:14:09.394 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.391+0000 7f88d59cf640 10 monclient: _renew_subs 2024-03-25T19:14:09.394 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.391+0000 7f88d59cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:09.394 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.391+0000 7f88d59cf640 1 -- 172.21.15.192:0/3031048156 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f88d00aa1a0 con 0x7f88d006c360 2024-03-25T19:14:09.394 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.391+0000 7f88d59cf640 1 librados: init done 2024-03-25T19:14:09.394 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.392+0000 7f88d59cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:09.394 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.392+0000 7f88d59cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:09.394 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.392+0000 7f88d59cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:09.394 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.392+0000 7f88d59cf640 1 -- 172.21.15.192:0/3031048156 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88d004f730 con 0x7f88d006c360 2024-03-25T19:14:09.449 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-03-25T19:14:09.449+0000 7f1c54213040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T19:14:09.489 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.488+0000 7f88b7fff640 1 -- 172.21.15.192:0/3031048156 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88c0003920 con 0x7f88d006c360 2024-03-25T19:14:09.489 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.488+0000 7f88b7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:14:09.489 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.488+0000 7f88b7fff640 1 -- 172.21.15.192:0/3031048156 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f88c0003cf0 con 0x7f88d006c360 2024-03-25T19:14:09.489 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.488+0000 7f88b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:09.489 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.488+0000 7f88b7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:14:09.489 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.488+0000 7f88b7fff640 10 monclient: dump: 2024-03-25T19:14:09.489 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:09.490 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:09.490 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:09.490 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:09.490 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:09.490 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:09.490 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:09.490 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:09.490 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:09.490 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:09.490 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.488+0000 7f88b7fff640 1 -- 172.21.15.192:0/3031048156 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f88c00042a0 con 0x7f88d006c360 2024-03-25T19:14:09.490 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.488+0000 7f88cd7fa640 4 set_mon_vals no callback set 2024-03-25T19:14:09.490 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.488+0000 7f88b7fff640 1 -- 172.21.15.192:0/3031048156 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 2249089290 0 0) 0x7f88c00068b0 con 0x7f88d006c360 2024-03-25T19:14:09.490 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.488+0000 7f88b7fff640 10 client.4147.objecter ms_dispatch 0x7f88d0000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T19:14:09.491 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.488+0000 7f88b7fff640 3 client.4147.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T19:14:09.491 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.488+0000 7f88b7fff640 3 client.4147.objecter handle_osd_map decoding full epoch 7 2024-03-25T19:14:09.491 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.488+0000 7f88b7fff640 20 client.4147.objecter dump_active .. 0 homeless 2024-03-25T19:14:09.491 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.488+0000 7f88b7fff640 1 -- 172.21.15.192:0/3031048156 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f88c0018de0 con 0x7f88d006c360 2024-03-25T19:14:09.491 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.488+0000 7f88b7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:09.491 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.488+0000 7f88b7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:14:09.560 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.560+0000 7f88d59cf640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "31abd8d9-93e5-4f4b-928e-d044c4b46999", "id": 5}] 2024-03-25T19:14:09.560 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.560+0000 7f88d59cf640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "31abd8d9-93e5-4f4b-928e-d044c4b46999", "id": 5}] 2024-03-25T19:14:09.560 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.560+0000 7f88d59cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:09.561 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.560+0000 7f88d59cf640 1 -- 172.21.15.192:0/3031048156 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd new", "uuid": "31abd8d9-93e5-4f4b-928e-d044c4b46999", "id": 5} v 0) v1 -- 0x7f88d006a040 con 0x7f88d006c360 2024-03-25T19:14:09.643 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.643+0000 7f88b7fff640 1 -- 172.21.15.192:0/3031048156 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "31abd8d9-93e5-4f4b-928e-d044c4b46999", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 3114817146 0 1988412176) 0x7f88c0006400 con 0x7f88d006c360 2024-03-25T19:14:09.644 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.643+0000 7f88b7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "31abd8d9-93e5-4f4b-928e-d044c4b46999", "id": 5}] 2024-03-25T19:14:09.644 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.643+0000 7f88b7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:14:09.644 INFO:teuthology.orchestra.run.smithi192.stdout:5 2024-03-25T19:14:09.645 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.644+0000 7f88d59cf640 10 librados: watch_flush enter 2024-03-25T19:14:09.645 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.644+0000 7f88d59cf640 10 librados: watch_flush exit 2024-03-25T19:14:09.645 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.644+0000 7f88d59cf640 20 client.4147.objecter shutdown clearing up homeless session... 2024-03-25T19:14:09.645 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.644+0000 7f88d59cf640 10 client.4147.objecter successfully canceled tick 2024-03-25T19:14:09.645 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.644+0000 7f88d59cf640 10 monclient: shutdown 2024-03-25T19:14:09.645 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.644+0000 7f88d59cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:14:09.646 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.644+0000 7f88d59cf640 1 -- 172.21.15.192:0/3031048156 >> v1:172.21.15.192:6789/0 conn(0x7f88d006c360 legacy=0x7f88d00aa980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:09.646 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.644+0000 7f88d59cf640 1 -- 172.21.15.192:0/3031048156 shutdown_connections 2024-03-25T19:14:09.646 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.644+0000 7f88d59cf640 1 -- 172.21.15.192:0/3031048156 >> 172.21.15.192:0/3031048156 conn(0x7f88d00b5ce0 msgr2=0x7f88d00b0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:14:09.646 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.644+0000 7f88d59cf640 1 -- 172.21.15.192:0/3031048156 shutdown_connections 2024-03-25T19:14:09.646 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.644+0000 7f88d59cf640 1 -- 172.21.15.192:0/3031048156 wait complete. 2024-03-25T19:14:09.646 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.644+0000 7f88d59cf640 1 librados: shutdown 2024-03-25T19:14:09.655 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new 861c9465-28f4-4980-9f1b-e411962ed333 6 2024-03-25T19:14:09.796 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.795+0000 7fe8f693d640 10 monclient: get_monmap_and_config 2024-03-25T19:14:09.796 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.795+0000 7fe8f693d640 10 monclient: build_initial_monmap 2024-03-25T19:14:09.796 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.795+0000 7fe8f693d640 10 monclient: monmap: 2024-03-25T19:14:09.796 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2024-03-25T19:14:09.796 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:09.796 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:14:09.796744+0000 2024-03-25T19:14:09.796 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:14:09.796744+0000 2024-03-25T19:14:09.796 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2024-03-25T19:14:09.796 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:09.796 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:14:09.796 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:14:09.797 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:14:09.797 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:09.798 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.796+0000 7fe8f693d640 1 Processor -- start 2024-03-25T19:14:09.798 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.796+0000 7fe8f693d640 1 -- start start 2024-03-25T19:14:09.798 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.796+0000 7fe8f693d640 10 monclient: init 2024-03-25T19:14:09.798 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.797+0000 7fe8f693d640 10 monclient: _reopen_session rank -1 2024-03-25T19:14:09.798 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.797+0000 7fe8f693d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:14:09.798 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.797+0000 7fe8f693d640 10 monclient(hunting): picked mon.noname-b con 0x7fe8f00b4b50 addr v1:172.21.15.6:6790/0 2024-03-25T19:14:09.798 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.797+0000 7fe8f693d640 10 monclient(hunting): picked mon.noname-c con 0x7fe8f013ff20 addr v1:172.21.15.192:6789/0 2024-03-25T19:14:09.798 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.797+0000 7fe8f693d640 10 monclient(hunting): picked mon.noname-a con 0x7fe8f0142dd0 addr v1:172.21.15.6:6789/0 2024-03-25T19:14:09.798 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.797+0000 7fe8f693d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8f00ab440 con 0x7fe8f0142dd0 2024-03-25T19:14:09.799 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.797+0000 7fe8f693d640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8f00ab6b0 con 0x7fe8f013ff20 2024-03-25T19:14:09.799 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.797+0000 7fe8f693d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8f00a9f10 con 0x7fe8f00b4b50 2024-03-25T19:14:09.799 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.797+0000 7fe8f693d640 10 monclient(hunting): _renew_subs 2024-03-25T19:14:09.799 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.797+0000 7fe8f693d640 10 monclient(hunting): authenticate will time out at 5972.090507s 2024-03-25T19:14:09.799 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.797+0000 7fe8f4eda640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe8f0142dd0 0x7fe8f014df50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.192:42872/0 (socket says 172.21.15.192:42872) 2024-03-25T19:14:09.799 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.797+0000 7fe8f4eda640 1 -- 172.21.15.192:0/2339019999 learned_addr learned my addr 172.21.15.192:0/2339019999 (peer_addr_for_me v1:172.21.15.192:0/0) 2024-03-25T19:14:09.800 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.799+0000 7fe8eeffd640 1 -- 172.21.15.192:0/2339019999 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1466867358 0 0) 0x7fe8f00ab6b0 con 0x7fe8f013ff20 2024-03-25T19:14:09.800 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.799+0000 7fe8eeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:14:09.800 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.799+0000 7fe8eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:09.800 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.799+0000 7fe8eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:09.800 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.799+0000 7fe8eeffd640 10 monclient(hunting): my global_id is 4150 2024-03-25T19:14:09.800 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.799+0000 7fe8eeffd640 1 -- 172.21.15.192:0/2339019999 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8d8003520 con 0x7fe8f013ff20 2024-03-25T19:14:09.800 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.800+0000 7fe8eeffd640 1 -- 172.21.15.192:0/2339019999 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2569128490 0 0) 0x7fe8f00ab440 con 0x7fe8f0142dd0 2024-03-25T19:14:09.800 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.800+0000 7fe8eeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:14:09.800 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.800+0000 7fe8eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:09.801 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.800+0000 7fe8eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:09.801 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.800+0000 7fe8eeffd640 10 monclient(hunting): my global_id is 4152 2024-03-25T19:14:09.801 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.800+0000 7fe8eeffd640 1 -- 172.21.15.192:0/2339019999 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8f00ab6b0 con 0x7fe8f0142dd0 2024-03-25T19:14:09.801 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.800+0000 7fe8eeffd640 1 -- 172.21.15.192:0/2339019999 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 671174824 0 0) 0x7fe8f00a9f10 con 0x7fe8f00b4b50 2024-03-25T19:14:09.801 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.800+0000 7fe8eeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:14:09.801 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.800+0000 7fe8eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:09.801 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.800+0000 7fe8eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:09.801 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.800+0000 7fe8eeffd640 10 monclient(hunting): my global_id is 4148 2024-03-25T19:14:09.801 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.800+0000 7fe8eeffd640 1 -- 172.21.15.192:0/2339019999 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8f00ab440 con 0x7fe8f00b4b50 2024-03-25T19:14:09.803 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.803+0000 7fe8eeffd640 1 -- 172.21.15.192:0/2339019999 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2870083317 0 0) 0x7fe8d8003520 con 0x7fe8f013ff20 2024-03-25T19:14:09.803 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.803+0000 7fe8eeffd640 1 -- 172.21.15.192:0/2339019999 >> v1:172.21.15.6:6790/0 conn(0x7fe8f00b4b50 legacy=0x7fe8f013f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:09.803 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.803+0000 7fe8eeffd640 1 -- 172.21.15.192:0/2339019999 >> v1:172.21.15.6:6789/0 conn(0x7fe8f0142dd0 legacy=0x7fe8f014df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:09.803 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.803+0000 7fe8eeffd640 10 monclient: _finish_hunting 0 2024-03-25T19:14:09.803 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.803+0000 7fe8eeffd640 1 monclient: found mon.noname-c 2024-03-25T19:14:09.803 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.803+0000 7fe8eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:14:09.803 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.803+0000 7fe8eeffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:14:09.804 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.803+0000 7fe8eeffd640 1 -- 172.21.15.192:0/2339019999 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f00aa0a0 con 0x7fe8f013ff20 2024-03-25T19:14:09.804 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.803+0000 7fe8eeffd640 10 monclient: _finish_auth 0 2024-03-25T19:14:09.804 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.803+0000 7fe8eeffd640 10 monclient: _check_auth_tickets 2024-03-25T19:14:09.804 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.803+0000 7fe8eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:14:09.804 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.803+0000 7fe8eeffd640 1 -- 172.21.15.192:0/2339019999 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fe8e0001690 con 0x7fe8f013ff20 2024-03-25T19:14:09.804 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.803+0000 7fe8f693d640 5 monclient: authenticate success, global_id 4150 2024-03-25T19:14:09.804 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.803+0000 7fe8f693d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:14:09.804 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.803+0000 7fe8eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:09.804 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.803+0000 7fe8eeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:14:09.804 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.803+0000 7fe8eeffd640 10 monclient: dump: 2024-03-25T19:14:09.804 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:09.804 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:09.804 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:09.805 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:09.805 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:09.805 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:09.805 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:09.805 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:09.805 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:09.805 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:09.805 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.803+0000 7fe8f693d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:14:09.808 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.804+0000 7fe8eeffd640 1 -- 172.21.15.192:0/2339019999 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8e00017f0 con 0x7fe8f013ff20 2024-03-25T19:14:09.808 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.804+0000 7fe8eeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:14:09.808 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.804+0000 7fe8eeffd640 1 -- 172.21.15.192:0/2339019999 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fe8e0001b80 con 0x7fe8f013ff20 2024-03-25T19:14:09.808 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.804+0000 7fe8eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:09.809 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.804+0000 7fe8eeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:14:09.809 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.804+0000 7fe8eeffd640 10 monclient: dump: 2024-03-25T19:14:09.809 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:09.809 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:09.809 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:09.809 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:09.809 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:09.809 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:09.809 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:09.809 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:09.809 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:09.809 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:09.809 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.804+0000 7fe8f693d640 10 monclient: get_monmap_and_config success 2024-03-25T19:14:09.809 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.804+0000 7fe8f693d640 4 set_mon_vals no callback set 2024-03-25T19:14:09.810 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.805+0000 7fe8f693d640 10 monclient: shutdown 2024-03-25T19:14:09.810 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.805+0000 7fe8f693d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:14:09.810 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.805+0000 7fe8f693d640 1 -- 172.21.15.192:0/2339019999 >> v1:172.21.15.192:6789/0 conn(0x7fe8f013ff20 legacy=0x7fe8f01426c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:09.810 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.805+0000 7fe8f693d640 1 -- 172.21.15.192:0/2339019999 shutdown_connections 2024-03-25T19:14:09.810 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.805+0000 7fe8f693d640 1 -- 172.21.15.192:0/2339019999 >> 172.21.15.192:0/2339019999 conn(0x7fe8f0066920 msgr2=0x7fe8f0066d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:14:09.810 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.805+0000 7fe8f693d640 1 -- 172.21.15.192:0/2339019999 shutdown_connections 2024-03-25T19:14:09.810 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.805+0000 7fe8f693d640 1 -- 172.21.15.192:0/2339019999 wait complete. 2024-03-25T19:14:09.810 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.806+0000 7fe8f693d640 10 monclient: build_initial_monmap 2024-03-25T19:14:09.810 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.806+0000 7fe8f693d640 10 monclient: monmap: 2024-03-25T19:14:09.810 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2024-03-25T19:14:09.810 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:14:09.810 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 0.000000 2024-03-25T19:14:09.810 INFO:teuthology.orchestra.run.smithi192.stderr:created 0.000000 2024-03-25T19:14:09.810 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2024-03-25T19:14:09.811 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2024-03-25T19:14:09.811 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:14:09.811 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:14:09.811 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:14:09.811 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:09.811 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.806+0000 7fe8f693d640 1 librados: starting msgr at 2024-03-25T19:14:09.811 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.806+0000 7fe8f693d640 1 librados: starting objecter 2024-03-25T19:14:09.811 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.806+0000 7fe8f693d640 1 Processor -- start 2024-03-25T19:14:09.811 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.806+0000 7fe8f693d640 1 -- start start 2024-03-25T19:14:09.811 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.806+0000 7fe8f693d640 1 librados: setting wanted keys 2024-03-25T19:14:09.811 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.806+0000 7fe8f693d640 1 librados: calling monclient init 2024-03-25T19:14:09.811 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.806+0000 7fe8f693d640 10 monclient: init 2024-03-25T19:14:09.811 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.807+0000 7fe8f693d640 10 monclient: _reopen_session rank -1 2024-03-25T19:14:09.811 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.807+0000 7fe8f693d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:14:09.811 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.807+0000 7fe8f693d640 10 monclient(hunting): picked mon.noname-c con 0x7fe8f00b4b50 addr v1:172.21.15.6:6790/0 2024-03-25T19:14:09.811 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.807+0000 7fe8f693d640 10 monclient(hunting): picked mon.noname-a con 0x7fe8f013ff20 addr v1:172.21.15.6:6789/0 2024-03-25T19:14:09.812 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.807+0000 7fe8f693d640 10 monclient(hunting): picked mon.noname-b con 0x7fe8f0142dd0 addr v1:172.21.15.192:6789/0 2024-03-25T19:14:09.812 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.807+0000 7fe8f693d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8f01439b0 con 0x7fe8f013ff20 2024-03-25T19:14:09.812 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.807+0000 7fe8f693d640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8f01422e0 con 0x7fe8f0142dd0 2024-03-25T19:14:09.812 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.807+0000 7fe8f693d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8f0142470 con 0x7fe8f00b4b50 2024-03-25T19:14:09.812 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.807+0000 7fe8f693d640 10 monclient(hunting): _renew_subs 2024-03-25T19:14:09.812 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.807+0000 7fe8f693d640 10 monclient(hunting): authenticate will time out at 5972.100754s 2024-03-25T19:14:09.812 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.807+0000 7fe8f4eda640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fe8f0142dd0 0x7fe8f01e26c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.192:58432/0 (socket says 172.21.15.192:58432) 2024-03-25T19:14:09.812 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.807+0000 7fe8f4eda640 1 -- 172.21.15.192:0/878252264 learned_addr learned my addr 172.21.15.192:0/878252264 (peer_addr_for_me v1:172.21.15.192:0/0) 2024-03-25T19:14:09.812 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.808+0000 7fe8ecff9640 10 client.?.objecter ms_handle_connect 0x7fe8f013ff20 2024-03-25T19:14:09.812 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.808+0000 7fe8ecff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:09.812 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.808+0000 7fe8ecff9640 10 client.?.objecter ms_handle_connect 0x7fe8f0142dd0 2024-03-25T19:14:09.812 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.808+0000 7fe8ecff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:09.812 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.808+0000 7fe8ecff9640 10 client.?.objecter ms_handle_connect 0x7fe8f00b4b50 2024-03-25T19:14:09.812 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.808+0000 7fe8ecff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:09.813 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.810+0000 7fe8ecff9640 1 -- 172.21.15.192:0/878252264 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2781657824 0 0) 0x7fe8f01439b0 con 0x7fe8f013ff20 2024-03-25T19:14:09.813 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.810+0000 7fe8ecff9640 10 monclient(hunting): _negotiate 2024-03-25T19:14:09.813 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.810+0000 7fe8ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:09.813 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.810+0000 7fe8ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:09.814 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.810+0000 7fe8ecff9640 10 monclient(hunting): my global_id is 4155 2024-03-25T19:14:09.814 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.810+0000 7fe8ecff9640 1 -- 172.21.15.192:0/878252264 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8dc004eb0 con 0x7fe8f013ff20 2024-03-25T19:14:09.814 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.810+0000 7fe8ecff9640 1 -- 172.21.15.192:0/878252264 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2527004373 0 0) 0x7fe8f01422e0 con 0x7fe8f0142dd0 2024-03-25T19:14:09.814 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.810+0000 7fe8ecff9640 10 monclient(hunting): _negotiate 2024-03-25T19:14:09.814 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.810+0000 7fe8ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:09.814 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.810+0000 7fe8ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:09.814 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.810+0000 7fe8ecff9640 10 monclient(hunting): my global_id is 4153 2024-03-25T19:14:09.814 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.810+0000 7fe8ecff9640 1 -- 172.21.15.192:0/878252264 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8f01439b0 con 0x7fe8f0142dd0 2024-03-25T19:14:09.814 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.810+0000 7fe8ecff9640 1 -- 172.21.15.192:0/878252264 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 831701407 0 0) 0x7fe8f0142470 con 0x7fe8f00b4b50 2024-03-25T19:14:09.814 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.810+0000 7fe8ecff9640 10 monclient(hunting): _negotiate 2024-03-25T19:14:09.814 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.810+0000 7fe8ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:09.814 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.810+0000 7fe8ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:09.814 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.810+0000 7fe8ecff9640 10 monclient(hunting): my global_id is 4151 2024-03-25T19:14:09.814 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.810+0000 7fe8ecff9640 1 -- 172.21.15.192:0/878252264 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8f01422e0 con 0x7fe8f00b4b50 2024-03-25T19:14:09.815 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.815+0000 7fe8ecff9640 1 -- 172.21.15.192:0/878252264 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 143850689 0 0) 0x7fe8f01439b0 con 0x7fe8f0142dd0 2024-03-25T19:14:09.815 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.815+0000 7fe8ecff9640 1 -- 172.21.15.192:0/878252264 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8f0142470 con 0x7fe8f0142dd0 2024-03-25T19:14:09.815 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.815+0000 7fe8ecff9640 1 -- 172.21.15.192:0/878252264 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fe8e8001c50 con 0x7fe8f0142dd0 2024-03-25T19:14:09.815 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.815+0000 7fe8ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:09.815 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.815+0000 7fe8ecff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:14:09.815 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.815+0000 7fe8ecff9640 10 monclient(hunting): dump: 2024-03-25T19:14:09.815 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:09.816 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:09.816 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:09.816 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:09.816 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:09.816 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:09.816 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:09.816 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:09.816 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:09.816 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:09.816 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.815+0000 7fe8ecff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:14:09.817 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.817+0000 7fe8ecff9640 1 -- 172.21.15.192:0/878252264 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 751214377 0 0) 0x7fe8dc004eb0 con 0x7fe8f013ff20 2024-03-25T19:14:09.817 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.817+0000 7fe8ecff9640 1 -- 172.21.15.192:0/878252264 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8f01439b0 con 0x7fe8f013ff20 2024-03-25T19:14:09.817 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.817+0000 7fe8ecff9640 1 -- 172.21.15.192:0/878252264 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fe8e0001b00 con 0x7fe8f013ff20 2024-03-25T19:14:09.817 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.817+0000 7fe8ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:09.817 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.817+0000 7fe8ecff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:14:09.817 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.817+0000 7fe8ecff9640 10 monclient(hunting): dump: 2024-03-25T19:14:09.817 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:09.818 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:09.818 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:09.818 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:09.818 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:09.818 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:09.818 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:09.818 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:09.818 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:09.818 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:09.818 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.817+0000 7fe8ecff9640 1 -- 172.21.15.192:0/878252264 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 544976949 0 0) 0x7fe8f01422e0 con 0x7fe8f00b4b50 2024-03-25T19:14:09.818 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.817+0000 7fe8ecff9640 1 -- 172.21.15.192:0/878252264 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8dc004eb0 con 0x7fe8f00b4b50 2024-03-25T19:14:09.818 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.817+0000 7fe8ecff9640 1 -- 172.21.15.192:0/878252264 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fe8e4001830 con 0x7fe8f00b4b50 2024-03-25T19:14:09.818 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.817+0000 7fe8ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:09.819 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.817+0000 7fe8ecff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:14:09.819 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.817+0000 7fe8ecff9640 10 monclient(hunting): dump: 2024-03-25T19:14:09.819 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:09.819 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:09.819 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:09.819 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:09.819 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:09.819 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:09.819 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:09.819 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:09.819 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:09.819 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:09.820 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.818+0000 7fe8ecff9640 1 -- 172.21.15.192:0/878252264 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1734499406 0 0) 0x7fe8f0142470 con 0x7fe8f0142dd0 2024-03-25T19:14:09.820 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.818+0000 7fe8ecff9640 1 -- 172.21.15.192:0/878252264 >> v1:172.21.15.6:6790/0 conn(0x7fe8f00b4b50 legacy=0x7fe8f01d7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:09.820 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.818+0000 7fe8ecff9640 1 -- 172.21.15.192:0/878252264 >> v1:172.21.15.6:6789/0 conn(0x7fe8f013ff20 legacy=0x7fe8f01d8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:09.821 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.818+0000 7fe8ecff9640 10 monclient: _finish_hunting 0 2024-03-25T19:14:09.821 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.818+0000 7fe8ecff9640 1 monclient: found mon.b 2024-03-25T19:14:09.821 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.818+0000 7fe8ecff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:14:09.821 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.818+0000 7fe8ecff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:09.821 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.818+0000 7fe8ecff9640 1 -- 172.21.15.192:0/878252264 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f0141740 con 0x7fe8f0142dd0 2024-03-25T19:14:09.821 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.818+0000 7fe8ecff9640 10 monclient: _finish_auth 0 2024-03-25T19:14:09.821 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.818+0000 7fe8ecff9640 10 monclient: _check_auth_tickets 2024-03-25T19:14:09.821 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.818+0000 7fe8ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:14:09.821 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.818+0000 7fe8f693d640 5 monclient: authenticate success, global_id 4153 2024-03-25T19:14:09.821 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.818+0000 7fe8f693d640 10 monclient: _renew_subs 2024-03-25T19:14:09.821 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.818+0000 7fe8f693d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:09.821 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.818+0000 7fe8f693d640 1 -- 172.21.15.192:0/878252264 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8f0141990 con 0x7fe8f0142dd0 2024-03-25T19:14:09.821 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.818+0000 7fe8f693d640 10 client.4153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:14:09.821 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.818+0000 7fe8f693d640 10 monclient: _renew_subs 2024-03-25T19:14:09.821 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.818+0000 7fe8f693d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:09.821 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.818+0000 7fe8f693d640 1 -- 172.21.15.192:0/878252264 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8f013e200 con 0x7fe8f0142dd0 2024-03-25T19:14:09.822 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.818+0000 7fe8f693d640 1 librados: init done 2024-03-25T19:14:09.822 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.819+0000 7fe8f693d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:09.822 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.819+0000 7fe8f693d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:09.822 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.819+0000 7fe8f693d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:09.822 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.819+0000 7fe8f693d640 1 -- 172.21.15.192:0/878252264 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8f01d7c80 con 0x7fe8f0142dd0 2024-03-25T19:14:09.926 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.926+0000 7fe8ecff9640 1 -- 172.21.15.192:0/878252264 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8e80020c0 con 0x7fe8f0142dd0 2024-03-25T19:14:09.926 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.926+0000 7fe8ecff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:14:09.927 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.926+0000 7fe8ecff9640 1 -- 172.21.15.192:0/878252264 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fe8e8004d00 con 0x7fe8f0142dd0 2024-03-25T19:14:09.927 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.926+0000 7fe8ecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:09.927 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.926+0000 7fe8ecff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:14:09.927 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.926+0000 7fe8ecff9640 10 monclient: dump: 2024-03-25T19:14:09.927 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:09.927 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:09.927 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:09.927 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:09.927 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:09.927 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:09.927 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:09.927 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:09.927 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:09.927 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:09.928 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.926+0000 7fe8ecff9640 1 -- 172.21.15.192:0/878252264 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe8e8005270 con 0x7fe8f0142dd0 2024-03-25T19:14:09.928 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.926+0000 7fe8ecff9640 1 -- 172.21.15.192:0/878252264 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 3564188894 0 0) 0x7fe8e8006a20 con 0x7fe8f0142dd0 2024-03-25T19:14:09.928 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.926+0000 7fe8ee7fc640 4 set_mon_vals no callback set 2024-03-25T19:14:09.928 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.926+0000 7fe8ecff9640 10 client.4153.objecter ms_dispatch 0x7fe8f0000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T19:14:09.928 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.926+0000 7fe8ecff9640 3 client.4153.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T19:14:09.928 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.926+0000 7fe8ecff9640 3 client.4153.objecter handle_osd_map decoding full epoch 8 2024-03-25T19:14:09.928 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.926+0000 7fe8ecff9640 20 client.4153.objecter dump_active .. 0 homeless 2024-03-25T19:14:09.928 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.926+0000 7fe8ecff9640 1 -- 172.21.15.192:0/878252264 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe8e8018de0 con 0x7fe8f0142dd0 2024-03-25T19:14:09.928 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.926+0000 7fe8ecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:09.928 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.926+0000 7fe8ecff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:14:09.998 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.997+0000 7fe8f693d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "861c9465-28f4-4980-9f1b-e411962ed333", "id": 6}] 2024-03-25T19:14:09.998 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.998+0000 7fe8f693d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "861c9465-28f4-4980-9f1b-e411962ed333", "id": 6}] 2024-03-25T19:14:09.998 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.998+0000 7fe8f693d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:09.998 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:09.998+0000 7fe8f693d640 1 -- 172.21.15.192:0/878252264 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd new", "uuid": "861c9465-28f4-4980-9f1b-e411962ed333", "id": 6} v 0) v1 -- 0x7fe8f0143b40 con 0x7fe8f0142dd0 2024-03-25T19:14:10.016 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-25T19:14:10.015+0000 7fbad1252040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T19:14:10.056 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.056+0000 7fe8ecff9640 1 -- 172.21.15.192:0/878252264 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "861c9465-28f4-4980-9f1b-e411962ed333", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 2849651762 0 1113811849) 0x7fe8e8006500 con 0x7fe8f0142dd0 2024-03-25T19:14:10.056 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.056+0000 7fe8ecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "861c9465-28f4-4980-9f1b-e411962ed333", "id": 6}] 2024-03-25T19:14:10.056 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.056+0000 7fe8ecff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:14:10.056 INFO:teuthology.orchestra.run.smithi192.stdout:6 2024-03-25T19:14:10.057 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.057+0000 7fe8f693d640 10 librados: watch_flush enter 2024-03-25T19:14:10.057 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.057+0000 7fe8f693d640 10 librados: watch_flush exit 2024-03-25T19:14:10.058 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.057+0000 7fe8f693d640 20 client.4153.objecter shutdown clearing up homeless session... 2024-03-25T19:14:10.058 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.057+0000 7fe8f693d640 10 client.4153.objecter successfully canceled tick 2024-03-25T19:14:10.058 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.057+0000 7fe8f693d640 10 monclient: shutdown 2024-03-25T19:14:10.058 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.057+0000 7fe8f693d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:14:10.058 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.057+0000 7fe8f693d640 1 -- 172.21.15.192:0/878252264 >> v1:172.21.15.192:6789/0 conn(0x7fe8f0142dd0 legacy=0x7fe8f01e26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:10.058 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.057+0000 7fe8f693d640 1 -- 172.21.15.192:0/878252264 shutdown_connections 2024-03-25T19:14:10.058 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.057+0000 7fe8f693d640 1 -- 172.21.15.192:0/878252264 >> 172.21.15.192:0/878252264 conn(0x7fe8f0066920 msgr2=0x7fe8f00a9660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:14:10.058 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.057+0000 7fe8f693d640 1 -- 172.21.15.192:0/878252264 shutdown_connections 2024-03-25T19:14:10.058 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.057+0000 7fe8f693d640 1 -- 172.21.15.192:0/878252264 wait complete. 2024-03-25T19:14:10.058 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.057+0000 7fe8f693d640 1 librados: shutdown 2024-03-25T19:14:10.068 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new 14bd0271-d543-4ea4-ab4c-61475195fc90 7 2024-03-25T19:14:10.183 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.183+0000 7f5d0d51c640 10 monclient: get_monmap_and_config 2024-03-25T19:14:10.184 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.183+0000 7f5d0d51c640 10 monclient: build_initial_monmap 2024-03-25T19:14:10.184 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.183+0000 7f5d0d51c640 10 monclient: monmap: 2024-03-25T19:14:10.184 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2024-03-25T19:14:10.184 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:10.184 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:14:10.184692+0000 2024-03-25T19:14:10.184 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:14:10.184692+0000 2024-03-25T19:14:10.184 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2024-03-25T19:14:10.184 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:10.184 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:14:10.184 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:14:10.184 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:14:10.184 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:10.186 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.184+0000 7f5d0d51c640 1 Processor -- start 2024-03-25T19:14:10.186 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.184+0000 7f5d0d51c640 1 -- start start 2024-03-25T19:14:10.186 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.184+0000 7f5d0d51c640 10 monclient: init 2024-03-25T19:14:10.186 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.184+0000 7f5d0d51c640 10 monclient: _reopen_session rank -1 2024-03-25T19:14:10.186 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.184+0000 7f5d0d51c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:14:10.186 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.184+0000 7f5d0d51c640 10 monclient(hunting): picked mon.noname-c con 0x7f5d0806c360 addr v1:172.21.15.192:6789/0 2024-03-25T19:14:10.187 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.184+0000 7f5d0d51c640 10 monclient(hunting): picked mon.noname-a con 0x7f5d0806c950 addr v1:172.21.15.6:6789/0 2024-03-25T19:14:10.187 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.184+0000 7f5d0d51c640 10 monclient(hunting): picked mon.noname-b con 0x7f5d080aee30 addr v1:172.21.15.6:6790/0 2024-03-25T19:14:10.187 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.184+0000 7f5d0d51c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d0806d5c0 con 0x7f5d0806c950 2024-03-25T19:14:10.187 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.184+0000 7f5d0d51c640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d080add40 con 0x7f5d0806c360 2024-03-25T19:14:10.187 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.184+0000 7f5d0d51c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d080ae020 con 0x7f5d080aee30 2024-03-25T19:14:10.187 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.185+0000 7f5d0d51c640 10 monclient(hunting): _renew_subs 2024-03-25T19:14:10.187 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.185+0000 7f5d0d51c640 10 monclient(hunting): authenticate will time out at 5972.478397s 2024-03-25T19:14:10.187 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.185+0000 7f5cfe7fc640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5d0806c950 0x7f5d080ae720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.192:42892/0 (socket says 172.21.15.192:42892) 2024-03-25T19:14:10.187 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.185+0000 7f5cfe7fc640 1 -- 172.21.15.192:0/3907264683 learned_addr learned my addr 172.21.15.192:0/3907264683 (peer_addr_for_me v1:172.21.15.192:0/0) 2024-03-25T19:14:10.188 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.187+0000 7f5d067fc640 1 -- 172.21.15.192:0/3907264683 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2936127550 0 0) 0x7f5d0806d5c0 con 0x7f5d0806c950 2024-03-25T19:14:10.188 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.187+0000 7f5d067fc640 10 monclient(hunting): _negotiate 2024-03-25T19:14:10.188 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.187+0000 7f5d067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:10.188 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.187+0000 7f5d067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:10.189 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.187+0000 7f5d067fc640 10 monclient(hunting): my global_id is 4158 2024-03-25T19:14:10.189 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.187+0000 7f5d067fc640 1 -- 172.21.15.192:0/3907264683 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cf4003520 con 0x7f5d0806c950 2024-03-25T19:14:10.189 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.188+0000 7f5d067fc640 1 -- 172.21.15.192:0/3907264683 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3228172876 0 0) 0x7f5d080add40 con 0x7f5d0806c360 2024-03-25T19:14:10.189 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.188+0000 7f5d067fc640 10 monclient(hunting): _negotiate 2024-03-25T19:14:10.189 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.188+0000 7f5d067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:10.189 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.188+0000 7f5d067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:10.189 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.188+0000 7f5d067fc640 10 monclient(hunting): my global_id is 4156 2024-03-25T19:14:10.189 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.188+0000 7f5d067fc640 1 -- 172.21.15.192:0/3907264683 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d0806d5c0 con 0x7f5d0806c360 2024-03-25T19:14:10.189 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.188+0000 7f5d067fc640 1 -- 172.21.15.192:0/3907264683 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3519412572 0 0) 0x7f5d080ae020 con 0x7f5d080aee30 2024-03-25T19:14:10.189 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.188+0000 7f5d067fc640 10 monclient(hunting): _negotiate 2024-03-25T19:14:10.189 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.188+0000 7f5d067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:10.189 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.188+0000 7f5d067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:10.190 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.188+0000 7f5d067fc640 10 monclient(hunting): my global_id is 4154 2024-03-25T19:14:10.190 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.188+0000 7f5d067fc640 1 -- 172.21.15.192:0/3907264683 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d080add40 con 0x7f5d080aee30 2024-03-25T19:14:10.191 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.191+0000 7f5d067fc640 1 -- 172.21.15.192:0/3907264683 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2162263928 0 0) 0x7f5cf4003520 con 0x7f5d0806c950 2024-03-25T19:14:10.191 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.191+0000 7f5d067fc640 1 -- 172.21.15.192:0/3907264683 >> v1:172.21.15.6:6790/0 conn(0x7f5d080aee30 legacy=0x7f5d0814e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:10.191 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.191+0000 7f5d067fc640 1 -- 172.21.15.192:0/3907264683 >> v1:172.21.15.192:6789/0 conn(0x7f5d0806c360 legacy=0x7f5d080b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:10.191 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.191+0000 7f5d067fc640 10 monclient: _finish_hunting 0 2024-03-25T19:14:10.191 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.191+0000 7f5d067fc640 1 monclient: found mon.noname-a 2024-03-25T19:14:10.191 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.191+0000 7f5d067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:14:10.191 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.191+0000 7f5d067fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-03-25T19:14:10.192 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.191+0000 7f5d067fc640 1 -- 172.21.15.192:0/3907264683 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d080ae300 con 0x7f5d0806c950 2024-03-25T19:14:10.192 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.191+0000 7f5d067fc640 10 monclient: _finish_auth 0 2024-03-25T19:14:10.192 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.191+0000 7f5d067fc640 10 monclient: _check_auth_tickets 2024-03-25T19:14:10.192 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.191+0000 7f5d067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:14:10.192 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.191+0000 7f5d067fc640 1 -- 172.21.15.192:0/3907264683 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f5cf0001690 con 0x7f5d0806c950 2024-03-25T19:14:10.192 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.191+0000 7f5d067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:10.192 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.191+0000 7f5d067fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:14:10.192 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.191+0000 7f5d067fc640 10 monclient: dump: 2024-03-25T19:14:10.192 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:10.192 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:10.192 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:10.192 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:10.192 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:10.192 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:10.193 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:10.193 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:10.193 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:10.193 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:10.193 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.191+0000 7f5d0d51c640 5 monclient: authenticate success, global_id 4158 2024-03-25T19:14:10.193 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.191+0000 7f5d0d51c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:14:10.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.192+0000 7f5d067fc640 1 -- 172.21.15.192:0/3907264683 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5cf00017f0 con 0x7f5d0806c950 2024-03-25T19:14:10.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.192+0000 7f5d067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:14:10.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.192+0000 7f5d067fc640 1 -- 172.21.15.192:0/3907264683 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f5cf0001c00 con 0x7f5d0806c950 2024-03-25T19:14:10.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.192+0000 7f5d0d51c640 10 monclient: get_monmap_and_config success 2024-03-25T19:14:10.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.192+0000 7f5d0d51c640 4 set_mon_vals no callback set 2024-03-25T19:14:10.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.192+0000 7f5d067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:10.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.192+0000 7f5d067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:14:10.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.192+0000 7f5d067fc640 10 monclient: dump: 2024-03-25T19:14:10.196 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:10.197 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:10.197 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:10.197 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:10.197 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:10.197 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:10.197 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:10.197 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:10.197 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:10.197 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:10.197 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.193+0000 7f5d0d51c640 10 monclient: shutdown 2024-03-25T19:14:10.197 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.193+0000 7f5d0d51c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:14:10.197 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.193+0000 7f5d0d51c640 1 -- 172.21.15.192:0/3907264683 >> v1:172.21.15.6:6789/0 conn(0x7f5d0806c950 legacy=0x7f5d080ae720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:10.197 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.193+0000 7f5d0d51c640 1 -- 172.21.15.192:0/3907264683 shutdown_connections 2024-03-25T19:14:10.197 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.193+0000 7f5d0d51c640 1 -- 172.21.15.192:0/3907264683 >> 172.21.15.192:0/3907264683 conn(0x7f5d080b5d50 msgr2=0x7f5d080b6130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:14:10.198 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.193+0000 7f5d0d51c640 1 -- 172.21.15.192:0/3907264683 shutdown_connections 2024-03-25T19:14:10.198 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.193+0000 7f5d0d51c640 1 -- 172.21.15.192:0/3907264683 wait complete. 2024-03-25T19:14:10.198 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.194+0000 7f5d0d51c640 10 monclient: build_initial_monmap 2024-03-25T19:14:10.198 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.194+0000 7f5d0d51c640 10 monclient: monmap: 2024-03-25T19:14:10.198 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2024-03-25T19:14:10.198 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:14:10.198 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 0.000000 2024-03-25T19:14:10.198 INFO:teuthology.orchestra.run.smithi192.stderr:created 0.000000 2024-03-25T19:14:10.198 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2024-03-25T19:14:10.198 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2024-03-25T19:14:10.198 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:14:10.198 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:14:10.198 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:14:10.199 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:10.199 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.194+0000 7f5d0d51c640 1 librados: starting msgr at 2024-03-25T19:14:10.199 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.194+0000 7f5d0d51c640 1 librados: starting objecter 2024-03-25T19:14:10.199 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.194+0000 7f5d0d51c640 1 Processor -- start 2024-03-25T19:14:10.199 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.194+0000 7f5d0d51c640 1 -- start start 2024-03-25T19:14:10.199 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.194+0000 7f5d0d51c640 1 librados: setting wanted keys 2024-03-25T19:14:10.199 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.194+0000 7f5d0d51c640 1 librados: calling monclient init 2024-03-25T19:14:10.199 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.194+0000 7f5d0d51c640 10 monclient: init 2024-03-25T19:14:10.199 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.195+0000 7f5d0d51c640 10 monclient: _reopen_session rank -1 2024-03-25T19:14:10.199 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.195+0000 7f5d0d51c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:14:10.199 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.195+0000 7f5d0d51c640 10 monclient(hunting): picked mon.noname-a con 0x7f5d0806c360 addr v1:172.21.15.6:6789/0 2024-03-25T19:14:10.199 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.195+0000 7f5d0d51c640 10 monclient(hunting): picked mon.noname-b con 0x7f5d0806c950 addr v1:172.21.15.192:6789/0 2024-03-25T19:14:10.199 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.195+0000 7f5d0d51c640 10 monclient(hunting): picked mon.noname-c con 0x7f5d080aee30 addr v1:172.21.15.6:6790/0 2024-03-25T19:14:10.199 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.195+0000 7f5d0d51c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d081a5110 con 0x7f5d0806c360 2024-03-25T19:14:10.200 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.195+0000 7f5d0d51c640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d081d80b0 con 0x7f5d0806c950 2024-03-25T19:14:10.200 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.195+0000 7f5d0d51c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d081d7d40 con 0x7f5d080aee30 2024-03-25T19:14:10.200 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.195+0000 7f5d0d51c640 10 monclient(hunting): _renew_subs 2024-03-25T19:14:10.200 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.195+0000 7f5d0d51c640 10 monclient(hunting): authenticate will time out at 5972.488863s 2024-03-25T19:14:10.200 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.195+0000 7f5d06ffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5d0806c360 0x7f5d081d7630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.192:42902/0 (socket says 172.21.15.192:42902) 2024-03-25T19:14:10.200 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.195+0000 7f5d06ffd640 1 -- 172.21.15.192:0/929159061 learned_addr learned my addr 172.21.15.192:0/929159061 (peer_addr_for_me v1:172.21.15.192:0/0) 2024-03-25T19:14:10.200 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.196+0000 7f5cfffff640 10 client.?.objecter ms_handle_connect 0x7f5d0806c360 2024-03-25T19:14:10.200 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.196+0000 7f5cfffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:10.200 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.196+0000 7f5cfffff640 10 client.?.objecter ms_handle_connect 0x7f5d0806c950 2024-03-25T19:14:10.200 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.196+0000 7f5cfffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:10.200 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.196+0000 7f5cfffff640 10 client.?.objecter ms_handle_connect 0x7f5d080aee30 2024-03-25T19:14:10.200 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.196+0000 7f5cfffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:10.201 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.198+0000 7f5cfffff640 1 -- 172.21.15.192:0/929159061 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 867859408 0 0) 0x7f5d081d80b0 con 0x7f5d0806c950 2024-03-25T19:14:10.201 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.198+0000 7f5cfffff640 10 monclient(hunting): _negotiate 2024-03-25T19:14:10.201 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.198+0000 7f5cfffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:10.201 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.198+0000 7f5cfffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:10.202 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.198+0000 7f5cfffff640 10 monclient(hunting): my global_id is 4159 2024-03-25T19:14:10.202 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.198+0000 7f5cfffff640 1 -- 172.21.15.192:0/929159061 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ce8004eb0 con 0x7f5d0806c950 2024-03-25T19:14:10.202 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.198+0000 7f5cfffff640 1 -- 172.21.15.192:0/929159061 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4292699915 0 0) 0x7f5d081a5110 con 0x7f5d0806c360 2024-03-25T19:14:10.202 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.198+0000 7f5cfffff640 10 monclient(hunting): _negotiate 2024-03-25T19:14:10.202 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.198+0000 7f5cfffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:10.202 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.198+0000 7f5cfffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:10.202 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.198+0000 7f5cfffff640 10 monclient(hunting): my global_id is 4161 2024-03-25T19:14:10.202 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.198+0000 7f5cfffff640 1 -- 172.21.15.192:0/929159061 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d081d80b0 con 0x7f5d0806c360 2024-03-25T19:14:10.202 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.199+0000 7f5cfffff640 1 -- 172.21.15.192:0/929159061 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3262143587 0 0) 0x7f5d081d7d40 con 0x7f5d080aee30 2024-03-25T19:14:10.202 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.199+0000 7f5cfffff640 10 monclient(hunting): _negotiate 2024-03-25T19:14:10.202 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.199+0000 7f5cfffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:10.202 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.199+0000 7f5cfffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:10.202 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.199+0000 7f5cfffff640 10 monclient(hunting): my global_id is 4157 2024-03-25T19:14:10.202 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.199+0000 7f5cfffff640 1 -- 172.21.15.192:0/929159061 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d081a5110 con 0x7f5d080aee30 2024-03-25T19:14:10.203 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.203+0000 7f5cfffff640 1 -- 172.21.15.192:0/929159061 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1882004367 0 0) 0x7f5ce8004eb0 con 0x7f5d0806c950 2024-03-25T19:14:10.203 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.203+0000 7f5cfffff640 1 -- 172.21.15.192:0/929159061 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d081d7d40 con 0x7f5d0806c950 2024-03-25T19:14:10.203 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.203+0000 7f5cfffff640 1 -- 172.21.15.192:0/929159061 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f5cf0001b00 con 0x7f5d0806c950 2024-03-25T19:14:10.204 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.203+0000 7f5cfffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:10.204 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.203+0000 7f5cfffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:14:10.204 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.203+0000 7f5cfffff640 10 monclient(hunting): dump: 2024-03-25T19:14:10.204 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:10.204 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:10.204 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:10.204 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:10.204 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:10.204 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:10.204 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:10.204 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:10.204 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:10.204 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:10.204 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.203+0000 7f5cfffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:14:10.204 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.203+0000 7f5cfffff640 1 -- 172.21.15.192:0/929159061 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3529956815 0 0) 0x7f5d081d80b0 con 0x7f5d0806c360 2024-03-25T19:14:10.204 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.203+0000 7f5cfffff640 1 -- 172.21.15.192:0/929159061 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ce8004eb0 con 0x7f5d0806c360 2024-03-25T19:14:10.205 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.203+0000 7f5cfffff640 1 -- 172.21.15.192:0/929159061 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f5cf80019b0 con 0x7f5d0806c360 2024-03-25T19:14:10.205 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.203+0000 7f5cfffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:10.205 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.203+0000 7f5cfffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:14:10.205 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.203+0000 7f5cfffff640 10 monclient(hunting): dump: 2024-03-25T19:14:10.205 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:10.205 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:10.205 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:10.205 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:10.205 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:10.205 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:10.205 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:10.205 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:10.205 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:10.205 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:10.207 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.205+0000 7f5cfffff640 1 -- 172.21.15.192:0/929159061 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 440473467 0 0) 0x7f5d081d7d40 con 0x7f5d0806c950 2024-03-25T19:14:10.207 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.205+0000 7f5cfffff640 1 -- 172.21.15.192:0/929159061 >> v1:172.21.15.6:6790/0 conn(0x7f5d080aee30 legacy=0x7f5d081a3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:10.207 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.205+0000 7f5cfffff640 1 -- 172.21.15.192:0/929159061 >> v1:172.21.15.6:6789/0 conn(0x7f5d0806c360 legacy=0x7f5d081d7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:10.207 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.205+0000 7f5cfffff640 10 monclient: _finish_hunting 0 2024-03-25T19:14:10.207 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.205+0000 7f5cfffff640 1 monclient: found mon.b 2024-03-25T19:14:10.207 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.205+0000 7f5cfffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:14:10.207 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.205+0000 7f5cfffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:10.207 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.205+0000 7f5cfffff640 1 -- 172.21.15.192:0/929159061 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d081c4820 con 0x7f5d0806c950 2024-03-25T19:14:10.207 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.205+0000 7f5cfffff640 10 monclient: _finish_auth 0 2024-03-25T19:14:10.207 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.205+0000 7f5cfffff640 10 monclient: _check_auth_tickets 2024-03-25T19:14:10.207 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.205+0000 7f5cfffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:14:10.207 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.205+0000 7f5d0d51c640 5 monclient: authenticate success, global_id 4159 2024-03-25T19:14:10.207 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.205+0000 7f5d0d51c640 10 monclient: _renew_subs 2024-03-25T19:14:10.207 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.205+0000 7f5d0d51c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:10.208 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.205+0000 7f5d0d51c640 1 -- 172.21.15.192:0/929159061 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d081c4ad0 con 0x7f5d0806c950 2024-03-25T19:14:10.208 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.205+0000 7f5d0d51c640 10 client.4159.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:14:10.208 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.205+0000 7f5d0d51c640 10 monclient: _renew_subs 2024-03-25T19:14:10.208 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.205+0000 7f5d0d51c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:10.208 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.205+0000 7f5d0d51c640 1 -- 172.21.15.192:0/929159061 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d081e3050 con 0x7f5d0806c950 2024-03-25T19:14:10.208 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.205+0000 7f5d0d51c640 1 librados: init done 2024-03-25T19:14:10.208 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.207+0000 7f5d0d51c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:10.208 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.207+0000 7f5d0d51c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:10.208 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.207+0000 7f5d0d51c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:10.208 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.207+0000 7f5d0d51c640 1 -- 172.21.15.192:0/929159061 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d08072990 con 0x7f5d0806c950 2024-03-25T19:14:10.208 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.208+0000 7f5cfffff640 1 -- 172.21.15.192:0/929159061 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5cf0001220 con 0x7f5d0806c950 2024-03-25T19:14:10.209 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.208+0000 7f5cfffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:14:10.209 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.208+0000 7f5d05ffb640 4 set_mon_vals no callback set 2024-03-25T19:14:10.305 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.305+0000 7f5cfffff640 1 -- 172.21.15.192:0/929159061 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f5cf0004a90 con 0x7f5d0806c950 2024-03-25T19:14:10.305 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.305+0000 7f5cfffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:10.305 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.305+0000 7f5cfffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:14:10.305 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.305+0000 7f5cfffff640 10 monclient: dump: 2024-03-25T19:14:10.305 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-03-25T19:14:10.305 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:10.306 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:10.306 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:10.306 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:10.306 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-03-25T19:14:10.306 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:10.306 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:10.306 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:10.306 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:14:10.306 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.305+0000 7f5cfffff640 1 -- 172.21.15.192:0/929159061 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f5cf0004f80 con 0x7f5d0806c950 2024-03-25T19:14:10.306 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.305+0000 7f5cfffff640 1 -- 172.21.15.192:0/929159061 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 1667777576 0 0) 0x7f5cf0006810 con 0x7f5d0806c950 2024-03-25T19:14:10.306 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.305+0000 7f5cfffff640 10 client.4159.objecter ms_dispatch 0x7f5d08000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T19:14:10.306 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.305+0000 7f5cfffff640 3 client.4159.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T19:14:10.306 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.305+0000 7f5cfffff640 3 client.4159.objecter handle_osd_map decoding full epoch 9 2024-03-25T19:14:10.306 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.305+0000 7f5cfffff640 20 client.4159.objecter dump_active .. 0 homeless 2024-03-25T19:14:10.307 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.305+0000 7f5cfffff640 1 -- 172.21.15.192:0/929159061 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f5cf0006bf0 con 0x7f5d0806c950 2024-03-25T19:14:10.307 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.305+0000 7f5cfffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:10.307 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.305+0000 7f5cfffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:14:10.326 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-03-25T19:14:10.326+0000 7f1c54213040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T19:14:10.334 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-25T19:14:10.334+0000 7fbad1252040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T19:14:10.376 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.375+0000 7f5d0d51c640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "14bd0271-d543-4ea4-ab4c-61475195fc90", "id": 7}] 2024-03-25T19:14:10.376 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.375+0000 7f5d0d51c640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "14bd0271-d543-4ea4-ab4c-61475195fc90", "id": 7}] 2024-03-25T19:14:10.376 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.376+0000 7f5d0d51c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:10.376 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.376+0000 7f5d0d51c640 1 -- 172.21.15.192:0/929159061 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd new", "uuid": "14bd0271-d543-4ea4-ab4c-61475195fc90", "id": 7} v 0) v1 -- 0x7f5d0804f730 con 0x7f5d0806c950 2024-03-25T19:14:10.388 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-25T19:14:10.388+0000 7fbad1252040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T19:14:10.426 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.426+0000 7f5cfffff640 1 -- 172.21.15.192:0/929159061 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "14bd0271-d543-4ea4-ab4c-61475195fc90", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 2366170097 0 1371665406) 0x7f5cf0006280 con 0x7f5d0806c950 2024-03-25T19:14:10.426 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.426+0000 7f5cfffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "14bd0271-d543-4ea4-ab4c-61475195fc90", "id": 7}] 2024-03-25T19:14:10.426 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.426+0000 7f5cfffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:14:10.426 INFO:teuthology.orchestra.run.smithi192.stdout:7 2024-03-25T19:14:10.427 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.426+0000 7f5d0d51c640 10 librados: watch_flush enter 2024-03-25T19:14:10.427 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.427+0000 7f5d0d51c640 10 librados: watch_flush exit 2024-03-25T19:14:10.428 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.427+0000 7f5d0d51c640 20 client.4159.objecter shutdown clearing up homeless session... 2024-03-25T19:14:10.428 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.427+0000 7f5d0d51c640 10 client.4159.objecter successfully canceled tick 2024-03-25T19:14:10.428 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.427+0000 7f5d0d51c640 10 monclient: shutdown 2024-03-25T19:14:10.428 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.427+0000 7f5d0d51c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:14:10.428 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.427+0000 7f5d0d51c640 1 -- 172.21.15.192:0/929159061 >> v1:172.21.15.192:6789/0 conn(0x7f5d0806c950 legacy=0x7f5d081a2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:10.428 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.427+0000 7f5d0d51c640 1 -- 172.21.15.192:0/929159061 shutdown_connections 2024-03-25T19:14:10.428 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.427+0000 7f5d0d51c640 1 -- 172.21.15.192:0/929159061 >> 172.21.15.192:0/929159061 conn(0x7f5d080b5d50 msgr2=0x7f5d081c9e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:14:10.428 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.427+0000 7f5d0d51c640 1 -- 172.21.15.192:0/929159061 shutdown_connections 2024-03-25T19:14:10.428 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.427+0000 7f5d0d51c640 1 -- 172.21.15.192:0/929159061 wait complete. 2024-03-25T19:14:10.428 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-25T19:14:10.427+0000 7f5d0d51c640 1 librados: shutdown 2024-03-25T19:14:10.437 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:14:10.437 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T19:14:10.438 DEBUG:teuthology.orchestra.run.smithi006:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T19:14:10.440 INFO:tasks.ceph.osd.0:Started 2024-03-25T19:14:10.440 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:14:10.440 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T19:14:10.440 DEBUG:teuthology.orchestra.run.smithi006:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T19:14:10.442 INFO:tasks.ceph.osd.1:Started 2024-03-25T19:14:10.442 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:14:10.442 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T19:14:10.442 DEBUG:teuthology.orchestra.run.smithi006:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T19:14:10.444 INFO:tasks.ceph.osd.2:Started 2024-03-25T19:14:10.444 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:14:10.444 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T19:14:10.445 DEBUG:teuthology.orchestra.run.smithi006:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T19:14:10.447 INFO:tasks.ceph.osd.3:Started 2024-03-25T19:14:10.447 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:14:10.447 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T19:14:10.447 DEBUG:teuthology.orchestra.run.smithi192:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T19:14:10.449 INFO:tasks.ceph.osd.4:Started 2024-03-25T19:14:10.449 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:14:10.449 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T19:14:10.449 DEBUG:teuthology.orchestra.run.smithi192:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T19:14:10.451 INFO:tasks.ceph.osd.5:Started 2024-03-25T19:14:10.451 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:14:10.452 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T19:14:10.452 DEBUG:teuthology.orchestra.run.smithi192:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T19:14:10.453 INFO:tasks.ceph.osd.6:Started 2024-03-25T19:14:10.454 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:14:10.454 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T19:14:10.454 DEBUG:teuthology.orchestra.run.smithi192:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T19:14:10.456 INFO:tasks.ceph.osd.7:Started 2024-03-25T19:14:10.456 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T19:14:10.560 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-25T19:14:10.560+0000 7fbad1252040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T19:14:10.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.583+0000 7f70e7faf640 10 monclient: get_monmap_and_config 2024-03-25T19:14:10.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.583+0000 7f70e7faf640 10 monclient: build_initial_monmap 2024-03-25T19:14:10.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.583+0000 7f70e7faf640 10 monclient: monmap: 2024-03-25T19:14:10.583 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-03-25T19:14:10.583 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:10.584 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-03-25T19:14:10.584613+0000 2024-03-25T19:14:10.584 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-03-25T19:14:10.584613+0000 2024-03-25T19:14:10.584 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:14:10.584 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:10.584 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:14:10.584 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:14:10.584 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:14:10.584 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-03-25T19:14:10.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.584+0000 7f70e7faf640 1 Processor -- start 2024-03-25T19:14:10.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.584+0000 7f70e7faf640 1 -- start start 2024-03-25T19:14:10.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.584+0000 7f70e7faf640 10 monclient: init 2024-03-25T19:14:10.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.585+0000 7f70e7faf640 10 monclient: _reopen_session rank -1 2024-03-25T19:14:10.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.585+0000 7f70e7faf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:14:10.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.585+0000 7f70e7faf640 10 monclient(hunting): picked mon.noname-b con 0x7f70e00b4f00 addr v1:172.21.15.6:6790/0 2024-03-25T19:14:10.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.585+0000 7f70e7faf640 10 monclient(hunting): picked mon.noname-c con 0x7f70e00b4510 addr v1:172.21.15.192:6789/0 2024-03-25T19:14:10.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.585+0000 7f70e7faf640 10 monclient(hunting): picked mon.noname-a con 0x7f70e0142ad0 addr v1:172.21.15.6:6789/0 2024-03-25T19:14:10.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.585+0000 7f70e7faf640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70e00a9d90 con 0x7f70e0142ad0 2024-03-25T19:14:10.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.585+0000 7f70e7faf640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70e00aa2b0 con 0x7f70e00b4510 2024-03-25T19:14:10.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.585+0000 7f70e7faf640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70e013fc60 con 0x7f70e00b4f00 2024-03-25T19:14:10.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.585+0000 7f70e7faf640 10 monclient(hunting): _renew_subs 2024-03-25T19:14:10.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.585+0000 7f70e7faf640 10 monclient(hunting): authenticate will time out at 5954.824941s 2024-03-25T19:14:10.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.585+0000 7f70e5d4b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f70e00b4f00 0x7f70e01423c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45722/0 (socket says 172.21.15.6:45722) 2024-03-25T19:14:10.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.585+0000 7f70e5d4b640 1 -- 172.21.15.6:0/2534921454 learned_addr learned my addr 172.21.15.6:0/2534921454 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:14:10.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.589+0000 7f70e4d49640 1 -- 172.21.15.6:0/2534921454 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 777376092 0 0) 0x7f70e013fc60 con 0x7f70e00b4f00 2024-03-25T19:14:10.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.589+0000 7f70e4d49640 10 monclient(hunting): _negotiate 2024-03-25T19:14:10.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.589+0000 7f70e4d49640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:10.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.589+0000 7f70e4d49640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:10.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.589+0000 7f70e4d49640 10 monclient(hunting): my global_id is 4160 2024-03-25T19:14:10.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.589+0000 7f70e4d49640 1 -- 172.21.15.6:0/2534921454 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70cc003520 con 0x7f70e00b4f00 2024-03-25T19:14:10.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.589+0000 7f70e4d49640 1 -- 172.21.15.6:0/2534921454 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3954357265 0 0) 0x7f70e00aa2b0 con 0x7f70e00b4510 2024-03-25T19:14:10.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.589+0000 7f70e4d49640 10 monclient(hunting): _negotiate 2024-03-25T19:14:10.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.589+0000 7f70e4d49640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:10.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.589+0000 7f70e4d49640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:10.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.589+0000 7f70e4d49640 10 monclient(hunting): my global_id is 4162 2024-03-25T19:14:10.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.589+0000 7f70e4d49640 1 -- 172.21.15.6:0/2534921454 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70e013fc60 con 0x7f70e00b4510 2024-03-25T19:14:10.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.593+0000 7f70e4d49640 1 -- 172.21.15.6:0/2534921454 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4253513889 0 0) 0x7f70cc003520 con 0x7f70e00b4f00 2024-03-25T19:14:10.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.593+0000 7f70e4d49640 1 -- 172.21.15.6:0/2534921454 >> v1:172.21.15.192:6789/0 conn(0x7f70e00b4510 legacy=0x7f70e013f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:10.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.593+0000 7f70e4d49640 1 -- 172.21.15.6:0/2534921454 >> v1:172.21.15.6:6789/0 conn(0x7f70e0142ad0 legacy=0x7f70e014dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:14:10.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.593+0000 7f70e4d49640 10 monclient: _finish_hunting 0 2024-03-25T19:14:10.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.593+0000 7f70e4d49640 1 monclient: found mon.noname-b 2024-03-25T19:14:10.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.593+0000 7f70e4d49640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:14:10.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.593+0000 7f70e4d49640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-03-25T19:14:10.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.593+0000 7f70e4d49640 1 -- 172.21.15.6:0/2534921454 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e013fdf0 con 0x7f70e00b4f00 2024-03-25T19:14:10.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.593+0000 7f70e4d49640 10 monclient: _finish_auth 0 2024-03-25T19:14:10.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.594+0000 7f70e4d49640 10 monclient: _check_auth_tickets 2024-03-25T19:14:10.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.594+0000 7f70e4d49640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:14:10.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.594+0000 7f70e4d49640 1 -- 172.21.15.6:0/2534921454 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f70dc001690 con 0x7f70e00b4f00 2024-03-25T19:14:10.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.594+0000 7f70e7faf640 5 monclient: authenticate success, global_id 4160 2024-03-25T19:14:10.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.594+0000 7f70e7faf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:14:10.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.594+0000 7f70e4d49640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:10.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.594+0000 7f70e4d49640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:14:10.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.594+0000 7f70e4d49640 10 monclient: dump: 2024-03-25T19:14:10.596 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-03-25T19:14:10.596 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:10.596 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:10.596 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:10.596 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:10.596 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:10.596 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:10.596 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:10.596 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:10.596 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-03-25T19:14:10.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.594+0000 7f70e7faf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:14:10.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.624+0000 7f70e4d49640 1 -- 172.21.15.6:0/2534921454 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70dc0018b0 con 0x7f70e00b4f00 2024-03-25T19:14:10.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.624+0000 7f70e4d49640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:14:10.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.624+0000 7f70e4d49640 1 -- 172.21.15.6:0/2534921454 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f70dc001c40 con 0x7f70e00b4f00 2024-03-25T19:14:10.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.624+0000 7f70e7faf640 10 monclient: get_monmap_and_config success 2024-03-25T19:14:10.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.624+0000 7f70e7faf640 4 set_mon_vals no callback set 2024-03-25T19:14:10.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.625+0000 7f70e4d49640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:10.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.625+0000 7f70e4d49640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:14:10.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.625+0000 7f70e4d49640 10 monclient: dump: 2024-03-25T19:14:10.625 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-03-25T19:14:10.625 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:10.625 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:10.625 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:10.625 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:10.625 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:10.626 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:10.626 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:10.626 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:10.626 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-03-25T19:14:10.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.625+0000 7f70e7faf640 10 monclient: shutdown 2024-03-25T19:14:10.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.625+0000 7f70e7faf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:14:10.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.625+0000 7f70e7faf640 1 -- 172.21.15.6:0/2534921454 >> v1:172.21.15.6:6790/0 conn(0x7f70e00b4f00 legacy=0x7f70e01423c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:10.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.625+0000 7f70e7faf640 1 -- 172.21.15.6:0/2534921454 shutdown_connections 2024-03-25T19:14:10.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.625+0000 7f70e7faf640 1 -- 172.21.15.6:0/2534921454 >> 172.21.15.6:0/2534921454 conn(0x7f70e0066b00 msgr2=0x7f70e0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:14:10.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.626+0000 7f70e7faf640 1 -- 172.21.15.6:0/2534921454 shutdown_connections 2024-03-25T19:14:10.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.627+0000 7f70e7faf640 1 -- 172.21.15.6:0/2534921454 wait complete. 2024-03-25T19:14:10.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.628+0000 7f70e7faf640 10 monclient: build_initial_monmap 2024-03-25T19:14:10.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.628+0000 7f70e7faf640 10 monclient: monmap: 2024-03-25T19:14:10.628 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-03-25T19:14:10.628 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:14:10.628 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-03-25T19:14:10.629 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-03-25T19:14:10.629 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:14:10.629 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-03-25T19:14:10.629 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:14:10.629 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:14:10.629 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:14:10.629 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-03-25T19:14:10.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.628+0000 7f70e7faf640 1 librados: starting msgr at 2024-03-25T19:14:10.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.628+0000 7f70e7faf640 1 librados: starting objecter 2024-03-25T19:14:10.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.628+0000 7f70e7faf640 1 Processor -- start 2024-03-25T19:14:10.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.628+0000 7f70e7faf640 1 -- start start 2024-03-25T19:14:10.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.628+0000 7f70e7faf640 1 librados: setting wanted keys 2024-03-25T19:14:10.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.628+0000 7f70e7faf640 1 librados: calling monclient init 2024-03-25T19:14:10.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.628+0000 7f70e7faf640 10 monclient: init 2024-03-25T19:14:10.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.629+0000 7f70e7faf640 10 monclient: _reopen_session rank -1 2024-03-25T19:14:10.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.629+0000 7f70e7faf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:14:10.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.629+0000 7f70e7faf640 10 monclient(hunting): picked mon.noname-b con 0x7f70e00b4510 addr v1:172.21.15.192:6789/0 2024-03-25T19:14:10.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.629+0000 7f70e7faf640 10 monclient(hunting): picked mon.noname-c con 0x7f70e00b4f00 addr v1:172.21.15.6:6790/0 2024-03-25T19:14:10.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.629+0000 7f70e7faf640 10 monclient(hunting): picked mon.noname-a con 0x7f70e0142ad0 addr v1:172.21.15.6:6789/0 2024-03-25T19:14:10.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.629+0000 7f70e7faf640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70e00ad2a0 con 0x7f70e0142ad0 2024-03-25T19:14:10.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.629+0000 7f70e7faf640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70e01da6f0 con 0x7f70e00b4510 2024-03-25T19:14:10.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.629+0000 7f70e7faf640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70e0146200 con 0x7f70e00b4f00 2024-03-25T19:14:10.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.629+0000 7f70e7faf640 10 monclient(hunting): _renew_subs 2024-03-25T19:14:10.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.629+0000 7f70e7faf640 10 monclient(hunting): authenticate will time out at 5954.869356s 2024-03-25T19:14:10.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.630+0000 7f70e5d4b640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f70e00b4510 0x7f70e00aba50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:55054/0 (socket says 172.21.15.6:55054) 2024-03-25T19:14:10.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.630+0000 7f70e5d4b640 1 -- 172.21.15.6:0/977406067 learned_addr learned my addr 172.21.15.6:0/977406067 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:14:10.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.630+0000 7f70d67fc640 10 client.?.objecter ms_handle_connect 0x7f70e00b4510 2024-03-25T19:14:10.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.630+0000 7f70d67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:10.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.631+0000 7f70d67fc640 10 client.?.objecter ms_handle_connect 0x7f70e00b4f00 2024-03-25T19:14:10.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.631+0000 7f70d67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:10.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.632+0000 7f70d67fc640 1 -- 172.21.15.6:0/977406067 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2435951672 0 0) 0x7f70e0146200 con 0x7f70e00b4f00 2024-03-25T19:14:10.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.632+0000 7f70d67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:14:10.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.632+0000 7f70d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:10.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.632+0000 7f70d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:10.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.632+0000 7f70d67fc640 10 monclient(hunting): my global_id is 4163 2024-03-25T19:14:10.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.632+0000 7f70d67fc640 1 -- 172.21.15.6:0/977406067 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70c4004eb0 con 0x7f70e00b4f00 2024-03-25T19:14:10.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.634+0000 7f70d67fc640 1 -- 172.21.15.6:0/977406067 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3008432780 0 0) 0x7f70e01da6f0 con 0x7f70e00b4510 2024-03-25T19:14:10.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.634+0000 7f70d67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:14:10.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.634+0000 7f70d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:10.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.634+0000 7f70d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:10.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.634+0000 7f70d67fc640 10 monclient(hunting): my global_id is 4165 2024-03-25T19:14:10.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.634+0000 7f70d67fc640 1 -- 172.21.15.6:0/977406067 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70e0146200 con 0x7f70e00b4510 2024-03-25T19:14:10.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.641+0000 7f70d67fc640 1 -- 172.21.15.6:0/977406067 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3278069686 0 0) 0x7f70e0146200 con 0x7f70e00b4510 2024-03-25T19:14:10.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.642+0000 7f70d67fc640 1 -- 172.21.15.6:0/977406067 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70e01da6f0 con 0x7f70e00b4510 2024-03-25T19:14:10.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.642+0000 7f70d67fc640 1 -- 172.21.15.6:0/977406067 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2534936716 0 0) 0x7f70c4004eb0 con 0x7f70e00b4f00 2024-03-25T19:14:10.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.642+0000 7f70d67fc640 1 -- 172.21.15.6:0/977406067 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70e0146200 con 0x7f70e00b4f00 2024-03-25T19:14:10.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.642+0000 7f70d67fc640 1 -- 172.21.15.6:0/977406067 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f70d0001930 con 0x7f70e00b4f00 2024-03-25T19:14:10.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.642+0000 7f70d67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:10.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.642+0000 7f70d67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:14:10.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.642+0000 7f70d67fc640 10 monclient(hunting): dump: 2024-03-25T19:14:10.643 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-03-25T19:14:10.643 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:10.644 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:10.644 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:10.644 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:10.644 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:10.644 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:10.644 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:10.644 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:10.644 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-03-25T19:14:10.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.642+0000 7f70d67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:14:10.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.642+0000 7f70d67fc640 10 client.?.objecter ms_handle_connect 0x7f70e0142ad0 2024-03-25T19:14:10.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.642+0000 7f70d67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:10.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.643+0000 7f70d67fc640 1 -- 172.21.15.6:0/977406067 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f70dc001730 con 0x7f70e00b4510 2024-03-25T19:14:10.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.643+0000 7f70d67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:10.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.643+0000 7f70d67fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:14:10.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.643+0000 7f70d67fc640 10 monclient(hunting): dump: 2024-03-25T19:14:10.645 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-03-25T19:14:10.645 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:10.645 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:10.645 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:10.645 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:10.645 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:10.645 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:10.645 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:10.645 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:10.645 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-03-25T19:14:10.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.646+0000 7f70d67fc640 1 -- 172.21.15.6:0/977406067 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4193127845 0 0) 0x7f70e0146200 con 0x7f70e00b4f00 2024-03-25T19:14:10.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.646+0000 7f70d67fc640 1 -- 172.21.15.6:0/977406067 >> v1:172.21.15.192:6789/0 conn(0x7f70e00b4510 legacy=0x7f70e00aba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:10.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.647+0000 7f70d67fc640 1 -- 172.21.15.6:0/977406067 >> v1:172.21.15.6:6789/0 conn(0x7f70e0142ad0 legacy=0x7f70e0138230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:10.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.647+0000 7f70d67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:14:10.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.647+0000 7f70d67fc640 1 monclient: found mon.c 2024-03-25T19:14:10.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.647+0000 7f70d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:14:10.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.647+0000 7f70d67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:14:10.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.647+0000 7f70d67fc640 1 -- 172.21.15.6:0/977406067 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e00ac160 con 0x7f70e00b4f00 2024-03-25T19:14:10.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.647+0000 7f70d67fc640 10 monclient: _finish_auth 0 2024-03-25T19:14:10.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.647+0000 7f70d67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:14:10.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.647+0000 7f70d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:14:10.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.647+0000 7f70e7faf640 5 monclient: authenticate success, global_id 4163 2024-03-25T19:14:10.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.647+0000 7f70e7faf640 10 monclient: _renew_subs 2024-03-25T19:14:10.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.647+0000 7f70e7faf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:14:10.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.647+0000 7f70e7faf640 1 -- 172.21.15.6:0/977406067 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70e01434b0 con 0x7f70e00b4f00 2024-03-25T19:14:10.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.648+0000 7f70e7faf640 10 client.4163.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:14:10.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.648+0000 7f70e7faf640 10 monclient: _renew_subs 2024-03-25T19:14:10.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.648+0000 7f70e7faf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:14:10.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.648+0000 7f70e7faf640 1 -- 172.21.15.6:0/977406067 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f70e01db500 con 0x7f70e00b4f00 2024-03-25T19:14:10.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.648+0000 7f70e7faf640 1 librados: init done 2024-03-25T19:14:10.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.649+0000 7f70e7faf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:10.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.649+0000 7f70e7faf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:10.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.649+0000 7f70e7faf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:14:10.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.649+0000 7f70e7faf640 1 -- 172.21.15.6:0/977406067 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70e004f730 con 0x7f70e00b4f00 2024-03-25T19:14:10.651 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-03-25T19:14:10.650+0000 7f1c54213040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T19:14:10.706 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-03-25T19:14:10.706+0000 7f1c54213040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T19:14:10.833 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-03-25T19:14:10.833+0000 7f1c54213040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T19:14:10.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.873+0000 7f70d67fc640 1 -- 172.21.15.6:0/977406067 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70d0001a90 con 0x7f70e00b4f00 2024-03-25T19:14:10.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.873+0000 7f70d67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:14:10.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.873+0000 7f70d67fc640 1 -- 172.21.15.6:0/977406067 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f70d00038a0 con 0x7f70e00b4f00 2024-03-25T19:14:10.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.873+0000 7f70d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:10.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.873+0000 7f70d67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:14:10.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.873+0000 7f70d67fc640 10 monclient: dump: 2024-03-25T19:14:10.874 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-03-25T19:14:10.874 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:10.874 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:10.874 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:10.874 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:10.874 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:10.874 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:10.874 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:10.874 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:10.874 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-03-25T19:14:10.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.873+0000 7f70d67fc640 1 -- 172.21.15.6:0/977406067 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f70d0003e50 con 0x7f70e00b4f00 2024-03-25T19:14:10.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.873+0000 7f70d67fc640 1 -- 172.21.15.6:0/977406067 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f70d0005720 con 0x7f70e00b4f00 2024-03-25T19:14:10.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.873+0000 7f70d67fc640 10 client.4163.objecter ms_dispatch 0x7f70e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:14:10.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.873+0000 7f70d67fc640 3 client.4163.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:14:10.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.873+0000 7f70d67fc640 3 client.4163.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:14:10.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.873+0000 7f70d7fff640 4 set_mon_vals no callback set 2024-03-25T19:14:10.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.873+0000 7f70d67fc640 20 client.4163.objecter dump_active .. 0 homeless 2024-03-25T19:14:10.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.873+0000 7f70d67fc640 1 -- 172.21.15.6:0/977406067 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f70d0005b40 con 0x7f70e00b4f00 2024-03-25T19:14:10.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.873+0000 7f70d67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:10.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.873+0000 7f70d67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:14:10.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.942+0000 7f70e7faf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:14:10.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.942+0000 7f70e7faf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:14:10.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.942+0000 7f70e7faf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:14:10.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:10.942+0000 7f70e7faf640 1 -- 172.21.15.6:0/977406067 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70e0075180 con 0x7f70e00b4f00 2024-03-25T19:14:11.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:11.172+0000 7f70d67fc640 1 -- 172.21.15.6:0/977406067 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 1694388279 0 0) 0x7f70d0005050 con 0x7f70e00b4f00 2024-03-25T19:14:11.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:11.314+0000 7f70d67fc640 1 -- 172.21.15.6:0/977406067 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f70d0015460 con 0x7f70e00b4f00 2024-03-25T19:14:11.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:11.314+0000 7f70d67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:14:11.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:11.314+0000 7f70d67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:14:11.314 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-25T19:14:11.314 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:14:11.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:11.315+0000 7f70e7faf640 10 librados: watch_flush enter 2024-03-25T19:14:11.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:11.315+0000 7f70e7faf640 10 librados: watch_flush exit 2024-03-25T19:14:11.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:11.315+0000 7f70e7faf640 20 client.4163.objecter shutdown clearing up homeless session... 2024-03-25T19:14:11.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:11.315+0000 7f70e7faf640 10 client.4163.objecter successfully canceled tick 2024-03-25T19:14:11.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:11.315+0000 7f70e7faf640 10 monclient: shutdown 2024-03-25T19:14:11.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:11.315+0000 7f70e7faf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:14:11.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:11.315+0000 7f70e7faf640 1 -- 172.21.15.6:0/977406067 >> v1:172.21.15.6:6790/0 conn(0x7f70e00b4f00 legacy=0x7f70e01dac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:11.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:11.315+0000 7f70e7faf640 1 -- 172.21.15.6:0/977406067 shutdown_connections 2024-03-25T19:14:11.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:11.315+0000 7f70e7faf640 1 -- 172.21.15.6:0/977406067 >> 172.21.15.6:0/977406067 conn(0x7f70e0066b00 msgr2=0x7f70e0141e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:14:11.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:11.315+0000 7f70e7faf640 1 -- 172.21.15.6:0/977406067 shutdown_connections 2024-03-25T19:14:11.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:11.315+0000 7f70e7faf640 1 -- 172.21.15.6:0/977406067 wait complete. 2024-03-25T19:14:11.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-25T19:14:11.315+0000 7f70e7faf640 1 librados: shutdown 2024-03-25T19:14:11.331 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T19:14:11.331 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T19:14:12.137 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:14:12.285 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.284+0000 7feef6c4a640 10 monclient: get_monmap_and_config 2024-03-25T19:14:12.285 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.284+0000 7feef6c4a640 10 monclient: build_initial_monmap 2024-03-25T19:14:12.285 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.284+0000 7feef6c4a640 10 monclient: monmap: 2024-03-25T19:14:12.285 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:14:12.285 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:12.285 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:14:12.285726+0000 2024-03-25T19:14:12.285 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:14:12.285726+0000 2024-03-25T19:14:12.285 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:14:12.285 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:12.285 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:14:12.285 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:14:12.285 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:14:12.286 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:12.287 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.285+0000 7feef6c4a640 1 Processor -- start 2024-03-25T19:14:12.287 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.285+0000 7feef6c4a640 1 -- start start 2024-03-25T19:14:12.287 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.285+0000 7feef6c4a640 10 monclient: init 2024-03-25T19:14:12.287 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.286+0000 7feef6c4a640 10 monclient: _reopen_session rank -1 2024-03-25T19:14:12.287 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.286+0000 7feef6c4a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:14:12.287 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.287+0000 7feef6c4a640 10 monclient(hunting): picked mon.noname-b con 0x7feef00b4ff0 addr v1:172.21.15.6:6790/0 2024-03-25T19:14:12.287 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.287+0000 7feef6c4a640 10 monclient(hunting): picked mon.noname-a con 0x7feef00b4670 addr v1:172.21.15.6:6789/0 2024-03-25T19:14:12.287 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.287+0000 7feef6c4a640 10 monclient(hunting): picked mon.noname-c con 0x7feef013c480 addr v1:172.21.15.192:6789/0 2024-03-25T19:14:12.288 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.287+0000 7feef6c4a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feef00ab0e0 con 0x7feef00b4670 2024-03-25T19:14:12.288 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.287+0000 7feef6c4a640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feef00ab3c0 con 0x7feef013c480 2024-03-25T19:14:12.288 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.287+0000 7feef6c4a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feef00a9f30 con 0x7feef00b4ff0 2024-03-25T19:14:12.288 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.287+0000 7feef6c4a640 10 monclient(hunting): _renew_subs 2024-03-25T19:14:12.288 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.287+0000 7feef6c4a640 10 monclient(hunting): authenticate will time out at 5956.527536s 2024-03-25T19:14:12.288 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.288+0000 7feef51e7640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7feef013c480 0x7feef014dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:55136/0 (socket says 172.21.15.6:55136) 2024-03-25T19:14:12.288 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.288+0000 7feef51e7640 1 -- 172.21.15.6:0/2566590217 learned_addr learned my addr 172.21.15.6:0/2566590217 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:14:12.483 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.483+0000 7feeef7fe640 1 -- 172.21.15.6:0/2566590217 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 716785626 0 0) 0x7feef00ab0e0 con 0x7feef00b4670 2024-03-25T19:14:12.483 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.483+0000 7feeef7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:14:12.484 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.483+0000 7feeef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:12.484 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.483+0000 7feeef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:12.484 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.483+0000 7feeef7fe640 10 monclient(hunting): my global_id is 4176 2024-03-25T19:14:12.484 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.483+0000 7feeef7fe640 1 -- 172.21.15.6:0/2566590217 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feee0003520 con 0x7feef00b4670 2024-03-25T19:14:12.533 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.533+0000 7feeef7fe640 1 -- 172.21.15.6:0/2566590217 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3333816293 0 0) 0x7feee0003520 con 0x7feef00b4670 2024-03-25T19:14:12.533 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.533+0000 7feeef7fe640 1 -- 172.21.15.6:0/2566590217 >> v1:172.21.15.6:6790/0 conn(0x7feef00b4ff0 legacy=0x7feef013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:12.533 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.533+0000 7feeef7fe640 1 -- 172.21.15.6:0/2566590217 >> v1:172.21.15.192:6789/0 conn(0x7feef013c480 legacy=0x7feef014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:12.533 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.533+0000 7feeef7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:14:12.533 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.533+0000 7feeef7fe640 1 monclient: found mon.noname-a 2024-03-25T19:14:12.534 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.533+0000 7feeef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:14:12.534 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.533+0000 7feeef7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-03-25T19:14:12.534 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.533+0000 7feeef7fe640 1 -- 172.21.15.6:0/2566590217 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef00a9b70 con 0x7feef00b4670 2024-03-25T19:14:12.534 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.533+0000 7feeef7fe640 10 monclient: _finish_auth 0 2024-03-25T19:14:12.534 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.533+0000 7feeef7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:14:12.534 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.533+0000 7feeef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:14:12.534 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.533+0000 7feef6c4a640 5 monclient: authenticate success, global_id 4176 2024-03-25T19:14:12.534 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.533+0000 7feef6c4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:14:12.535 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.536+0000 7feeef7fe640 1 -- 172.21.15.6:0/2566590217 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7feedc001690 con 0x7feef00b4670 2024-03-25T19:14:12.536 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.536+0000 7feeef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:12.536 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.536+0000 7feeef7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:14:12.536 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.536+0000 7feeef7fe640 10 monclient: dump: 2024-03-25T19:14:12.536 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:14:12.536 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:12.536 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:12.536 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:12.536 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:12.536 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:12.536 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:12.536 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:12.536 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:12.536 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:12.536 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.536+0000 7feef6c4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:14:12.554 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.554+0000 7feeef7fe640 1 -- 172.21.15.6:0/2566590217 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feedc001920 con 0x7feef00b4670 2024-03-25T19:14:12.554 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.554+0000 7feeef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:14:12.554 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.554+0000 7feeef7fe640 1 -- 172.21.15.6:0/2566590217 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7feedc001c30 con 0x7feef00b4670 2024-03-25T19:14:12.554 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.554+0000 7feeef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:12.554 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.554+0000 7feeef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:14:12.554 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.554+0000 7feeef7fe640 10 monclient: dump: 2024-03-25T19:14:12.554 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:14:12.555 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:12.555 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:12.555 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:12.555 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:12.555 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:12.555 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:12.555 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:12.555 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:12.555 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:12.555 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.554+0000 7feef6c4a640 10 monclient: get_monmap_and_config success 2024-03-25T19:14:12.555 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.554+0000 7feef6c4a640 4 set_mon_vals no callback set 2024-03-25T19:14:12.556 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.555+0000 7feef6c4a640 10 monclient: shutdown 2024-03-25T19:14:12.556 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.555+0000 7feef6c4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:14:12.556 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.555+0000 7feef6c4a640 1 -- 172.21.15.6:0/2566590217 >> v1:172.21.15.6:6789/0 conn(0x7feef00b4670 legacy=0x7feef01425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:12.556 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.555+0000 7feef6c4a640 1 -- 172.21.15.6:0/2566590217 shutdown_connections 2024-03-25T19:14:12.556 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.555+0000 7feef6c4a640 1 -- 172.21.15.6:0/2566590217 >> 172.21.15.6:0/2566590217 conn(0x7feef00617a0 msgr2=0x7feef0061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:14:12.556 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.555+0000 7feef6c4a640 1 -- 172.21.15.6:0/2566590217 shutdown_connections 2024-03-25T19:14:12.556 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.555+0000 7feef6c4a640 1 -- 172.21.15.6:0/2566590217 wait complete. 2024-03-25T19:14:12.557 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.556+0000 7feef6c4a640 10 monclient: build_initial_monmap 2024-03-25T19:14:12.557 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.556+0000 7feef6c4a640 10 monclient: monmap: 2024-03-25T19:14:12.557 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:14:12.557 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:14:12.557 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:14:12.557 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:14:12.558 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:14:12.558 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:14:12.558 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:14:12.558 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:14:12.558 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:14:12.558 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:12.558 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.557+0000 7feef6c4a640 1 librados: starting msgr at 2024-03-25T19:14:12.558 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.557+0000 7feef6c4a640 1 librados: starting objecter 2024-03-25T19:14:12.558 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.557+0000 7feef6c4a640 1 Processor -- start 2024-03-25T19:14:12.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.558+0000 7feef6c4a640 1 -- start start 2024-03-25T19:14:12.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.558+0000 7feef6c4a640 1 librados: setting wanted keys 2024-03-25T19:14:12.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.558+0000 7feef6c4a640 1 librados: calling monclient init 2024-03-25T19:14:12.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.558+0000 7feef6c4a640 10 monclient: init 2024-03-25T19:14:12.560 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.559+0000 7feef6c4a640 10 monclient: _reopen_session rank -1 2024-03-25T19:14:12.560 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.559+0000 7feef6c4a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:14:12.560 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.559+0000 7feef6c4a640 10 monclient(hunting): picked mon.noname-b con 0x7feef00b4670 addr v1:172.21.15.192:6789/0 2024-03-25T19:14:12.560 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.559+0000 7feef6c4a640 10 monclient(hunting): picked mon.noname-a con 0x7feef00b4ff0 addr v1:172.21.15.6:6789/0 2024-03-25T19:14:12.561 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.560+0000 7feef6c4a640 10 monclient(hunting): picked mon.noname-c con 0x7feef013c480 addr v1:172.21.15.6:6790/0 2024-03-25T19:14:12.561 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.560+0000 7feef6c4a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feef01452a0 con 0x7feef00b4ff0 2024-03-25T19:14:12.561 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.560+0000 7feef6c4a640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feef013efe0 con 0x7feef00b4670 2024-03-25T19:14:12.561 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.560+0000 7feef6c4a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feef0144930 con 0x7feef013c480 2024-03-25T19:14:12.561 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.560+0000 7feeeffff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7feef00b4ff0 0x7feef0145e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44828/0 (socket says 172.21.15.6:44828) 2024-03-25T19:14:12.561 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.560+0000 7feeeffff640 1 -- 172.21.15.6:0/3895669925 learned_addr learned my addr 172.21.15.6:0/3895669925 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:14:12.561 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.560+0000 7feef6c4a640 10 monclient(hunting): _renew_subs 2024-03-25T19:14:12.561 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.560+0000 7feef6c4a640 10 monclient(hunting): authenticate will time out at 5956.800536s 2024-03-25T19:14:12.561 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.561+0000 7feeed7fa640 10 client.?.objecter ms_handle_connect 0x7feef00b4ff0 2024-03-25T19:14:12.561 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.561+0000 7feeed7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:12.564 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.564+0000 7feeed7fa640 1 -- 172.21.15.6:0/3895669925 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 34222416 0 0) 0x7feef01452a0 con 0x7feef00b4ff0 2024-03-25T19:14:12.564 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.564+0000 7feeed7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:14:12.564 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.564+0000 7feeed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:12.564 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.564+0000 7feeed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:12.565 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.564+0000 7feeed7fa640 10 monclient(hunting): my global_id is 4179 2024-03-25T19:14:12.565 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.564+0000 7feeed7fa640 1 -- 172.21.15.6:0/3895669925 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feed4004eb0 con 0x7feef00b4ff0 2024-03-25T19:14:12.572 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.572+0000 7feeed7fa640 1 -- 172.21.15.6:0/3895669925 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2404472789 0 0) 0x7feed4004eb0 con 0x7feef00b4ff0 2024-03-25T19:14:12.572 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.572+0000 7feeed7fa640 1 -- 172.21.15.6:0/3895669925 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feef01452a0 con 0x7feef00b4ff0 2024-03-25T19:14:12.574 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.573+0000 7feeed7fa640 1 -- 172.21.15.6:0/3895669925 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7feedc0036b0 con 0x7feef00b4ff0 2024-03-25T19:14:12.574 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.573+0000 7feeed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:12.574 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.573+0000 7feeed7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:14:12.574 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.573+0000 7feeed7fa640 10 monclient(hunting): dump: 2024-03-25T19:14:12.574 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:14:12.574 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:12.574 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:12.574 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:12.574 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:12.574 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:12.574 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:12.575 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:12.575 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:12.575 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:12.575 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.573+0000 7feeed7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:14:12.579 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.579+0000 7feeed7fa640 1 -- 172.21.15.6:0/3895669925 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2896888546 0 0) 0x7feef01452a0 con 0x7feef00b4ff0 2024-03-25T19:14:12.580 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.579+0000 7feeed7fa640 1 -- 172.21.15.6:0/3895669925 >> v1:172.21.15.6:6790/0 conn(0x7feef013c480 legacy=0x7feef01e2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:12.580 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.579+0000 7feeed7fa640 1 -- 172.21.15.6:0/3895669925 >> v1:172.21.15.192:6789/0 conn(0x7feef00b4670 legacy=0x7feef0145710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:12.580 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.579+0000 7feeed7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:14:12.580 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.579+0000 7feeed7fa640 1 monclient: found mon.a 2024-03-25T19:14:12.580 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.579+0000 7feeed7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:14:12.580 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.579+0000 7feeed7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:14:12.580 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.579+0000 7feeed7fa640 1 -- 172.21.15.6:0/3895669925 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef0144ba0 con 0x7feef00b4ff0 2024-03-25T19:14:12.580 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.579+0000 7feeed7fa640 10 monclient: _finish_auth 0 2024-03-25T19:14:12.580 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.579+0000 7feeed7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:14:12.580 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.579+0000 7feeed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:14:12.581 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.580+0000 7feef6c4a640 5 monclient: authenticate success, global_id 4179 2024-03-25T19:14:12.581 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.580+0000 7feef6c4a640 10 monclient: _renew_subs 2024-03-25T19:14:12.581 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.580+0000 7feef6c4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:14:12.581 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.580+0000 7feef6c4a640 1 -- 172.21.15.6:0/3895669925 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feef0144e80 con 0x7feef00b4ff0 2024-03-25T19:14:12.581 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.580+0000 7feef6c4a640 10 client.4179.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:14:12.581 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.580+0000 7feef6c4a640 10 monclient: _renew_subs 2024-03-25T19:14:12.581 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.580+0000 7feef6c4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:14:12.581 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.580+0000 7feef6c4a640 1 -- 172.21.15.6:0/3895669925 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feef01438d0 con 0x7feef00b4ff0 2024-03-25T19:14:12.581 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.580+0000 7feef6c4a640 1 librados: init done 2024-03-25T19:14:12.582 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.581+0000 7feef6c4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:12.582 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.581+0000 7feef6c4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:12.582 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.581+0000 7feef6c4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:14:12.582 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.581+0000 7feef6c4a640 1 -- 172.21.15.6:0/3895669925 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feef0145430 con 0x7feef00b4ff0 2024-03-25T19:14:12.582 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.582+0000 7feeed7fa640 1 -- 172.21.15.6:0/3895669925 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feedc0011f0 con 0x7feef00b4ff0 2024-03-25T19:14:12.582 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.582+0000 7feeed7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:14:12.582 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.582+0000 7feeeeffd640 4 set_mon_vals no callback set 2024-03-25T19:14:12.591 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-25T19:14:12.591+0000 7fba71f91640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T19:14:12.613 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.614+0000 7feeed7fa640 1 -- 172.21.15.6:0/3895669925 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7feedc003c00 con 0x7feef00b4ff0 2024-03-25T19:14:12.614 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.614+0000 7feeed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:12.614 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.614+0000 7feeed7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:14:12.614 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.614+0000 7feeed7fa640 10 monclient: dump: 2024-03-25T19:14:12.614 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:14:12.614 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:12.614 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:12.614 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:12.614 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:12.614 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:12.614 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:12.614 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:12.614 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:12.614 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:12.615 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.615+0000 7feeed7fa640 1 -- 172.21.15.6:0/3895669925 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 723271067 0 0) 0x7feedc003e60 con 0x7feef00b4ff0 2024-03-25T19:14:12.616 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.616+0000 7feef49e6640 10 monclient: get_auth_request con 0x7feed406d910 auth_method 0 2024-03-25T19:14:12.616 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:12.616+0000 7feeed7fa640 10 client.4179.objecter ms_handle_connect 0x7feed406d910 2024-03-25T19:14:12.732 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-25T19:14:12.733+0000 7fba71f91640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T19:14:13.055 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:13.055+0000 7feeed7fa640 1 -- 172.21.15.6:0/3895669925 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7feedc054980 con 0x7feef00b4ff0 2024-03-25T19:14:13.056 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:13.055+0000 7feeed7fa640 10 client.4179.objecter ms_dispatch 0x7feef0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:14:13.056 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:13.055+0000 7feeed7fa640 3 client.4179.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:14:13.056 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:13.055+0000 7feeed7fa640 3 client.4179.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:14:13.056 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:13.055+0000 7feeed7fa640 20 client.4179.objecter dump_active .. 0 homeless 2024-03-25T19:14:13.056 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:13.055+0000 7feeed7fa640 1 -- 172.21.15.6:0/3895669925 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feedc050490 con 0x7feef00b4ff0 2024-03-25T19:14:13.056 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:13.055+0000 7feeed7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:13.056 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:13.055+0000 7feeed7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:14:13.425 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:13.424+0000 7feef6c4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:14:13.425 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:13.424+0000 7feef6c4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:14:13.425 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:13.424+0000 7feef6c4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:14:13.425 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:13.424+0000 7feef6c4a640 1 -- 172.21.15.6:0/3895669925 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feef013ee10 con 0x7feef00b4ff0 2024-03-25T19:14:13.531 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:13.531+0000 7feeed7fa640 1 -- 172.21.15.6:0/3895669925 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7feedc0523c0 con 0x7feef00b4ff0 2024-03-25T19:14:13.531 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:13.531+0000 7feeed7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:14:13.531 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:13.531+0000 7feeed7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:14:13.531 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:14:13.531 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:14:13.535 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:13.534+0000 7feef6c4a640 10 librados: watch_flush enter 2024-03-25T19:14:13.535 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:13.534+0000 7feef6c4a640 10 librados: watch_flush exit 2024-03-25T19:14:13.535 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:13.534+0000 7feef6c4a640 20 client.4179.objecter shutdown clearing up homeless session... 2024-03-25T19:14:13.535 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:13.534+0000 7feef6c4a640 10 client.4179.objecter successfully canceled tick 2024-03-25T19:14:13.535 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:13.534+0000 7feef6c4a640 1 -- 172.21.15.6:0/3895669925 >> v1:172.21.15.6:6800/2936652650 conn(0x7feed406d910 legacy=0x7feed406fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:13.535 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:13.534+0000 7feef6c4a640 10 monclient: shutdown 2024-03-25T19:14:13.535 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:13.534+0000 7feef6c4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:14:13.535 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:13.534+0000 7feef6c4a640 1 -- 172.21.15.6:0/3895669925 >> v1:172.21.15.6:6789/0 conn(0x7feef00b4ff0 legacy=0x7feef0145e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:13.535 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:13.534+0000 7feef6c4a640 1 -- 172.21.15.6:0/3895669925 shutdown_connections 2024-03-25T19:14:13.535 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:13.534+0000 7feef6c4a640 1 -- 172.21.15.6:0/3895669925 >> 172.21.15.6:0/3895669925 conn(0x7feef00617a0 msgr2=0x7feef01407a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:14:13.535 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:13.534+0000 7feef6c4a640 1 -- 172.21.15.6:0/3895669925 shutdown_connections 2024-03-25T19:14:13.535 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:13.534+0000 7feef6c4a640 1 -- 172.21.15.6:0/3895669925 wait complete. 2024-03-25T19:14:13.536 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:13.534+0000 7feef6c4a640 1 librados: shutdown 2024-03-25T19:14:13.547 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:14:17.463 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:06.944 35208== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:17.513 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:06.979 35211== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:17.517 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:06.998 35209== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:17.524 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:07.005 35208== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:17.579 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:07.060 35209== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:17.584 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:07.050 35211== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:17.646 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:07.125 35210== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:17.703 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:07.183 35210== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:17.815 INFO:tasks.ceph.osd.1.smithi006.stderr:==00:00:00:07.283 35454== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:17.882 INFO:tasks.ceph.osd.1.smithi006.stderr:==00:00:00:07.351 35454== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:17.917 INFO:tasks.ceph.osd.2.smithi006.stderr:==00:00:00:07.401 35452== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:17.978 INFO:tasks.ceph.osd.2.smithi006.stderr:==00:00:00:07.462 35452== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:18.081 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-03-25T19:14:18.073+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:14:18.115 INFO:tasks.ceph.osd.0.smithi006.stderr:==00:00:00:07.607 35451== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:18.129 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-03-25T19:14:18.123+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:14:18.151 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-03-25T19:14:18.143+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:14:18.175 INFO:tasks.ceph.osd.0.smithi006.stderr:==00:00:00:07.667 35451== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:18.285 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-03-25T19:14:18.279+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:14:18.334 INFO:tasks.ceph.osd.3.smithi006.stderr:==00:00:00:07.804 35453== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:18.395 INFO:tasks.ceph.osd.3.smithi006.stderr:==00:00:00:07.864 35453== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:18.429 INFO:tasks.ceph.osd.1.smithi006.stderr:2024-03-25T19:14:18.423+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:14:18.534 INFO:tasks.ceph.osd.2.smithi006.stderr:2024-03-25T19:14:18.528+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:14:18.576 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:08.057 35208== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:18.598 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:08.079 35209== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:18.621 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:08.086 35211== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:18.747 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-03-25T19:14:18.741+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:14:18.753 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:08.233 35210== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:18.897 INFO:tasks.ceph.osd.1.smithi006.stderr:==00:00:00:08.366 35454== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:18.940 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:08.422 35208== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:18.972 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:08.453 35209== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:18.978 INFO:tasks.ceph.osd.3.smithi006.stderr:2024-03-25T19:14:18.970+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:14:19.014 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:08.480 35211== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:19.020 INFO:tasks.ceph.osd.2.smithi006.stderr:==00:00:00:08.505 35452== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:19.120 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:08.599 35210== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:19.214 INFO:tasks.ceph.osd.0.smithi006.stderr:==00:00:00:08.707 35451== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:19.268 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:08.749 35208== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:19.290 INFO:tasks.ceph.osd.1.smithi006.stderr:==00:00:00:08.759 35454== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:19.294 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:08.776 35209== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:19.372 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:08.838 35211== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:19.407 INFO:tasks.ceph.osd.2.smithi006.stderr:==00:00:00:08.891 35452== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:19.440 INFO:tasks.ceph.osd.3.smithi006.stderr:==00:00:00:08.909 35453== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:19.471 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:08.951 35210== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:19.535 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:09.016 35208== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:19.560 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:09.041 35209== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:19.581 INFO:tasks.ceph.osd.0.smithi006.stderr:==00:00:00:09.074 35451== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:19.659 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:09.125 35211== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:19.663 INFO:tasks.ceph.osd.1.smithi006.stderr:==00:00:00:09.132 35454== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:19.764 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:09.243 35210== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:19.764 INFO:tasks.ceph.osd.2.smithi006.stderr:==00:00:00:09.248 35452== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:19.808 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:09.289 35208== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:19.822 INFO:tasks.ceph.osd.3.smithi006.stderr:==00:00:00:09.292 35453== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:19.830 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:09.312 35209== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:19.947 INFO:tasks.ceph.osd.0.smithi006.stderr:==00:00:00:09.440 35451== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:19.950 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:09.415 35211== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:19.953 INFO:tasks.ceph.osd.1.smithi006.stderr:==00:00:00:09.421 35454== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:20.053 INFO:tasks.ceph.osd.2.smithi006.stderr:==00:00:00:09.537 35452== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:20.059 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:09.538 35210== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:20.081 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:09.563 35208== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:20.104 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:09.586 35209== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:20.141 INFO:tasks.ceph.osd.3.smithi006.stderr:==00:00:00:09.610 35453== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:20.241 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:09.707 35211== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:20.243 INFO:tasks.ceph.osd.0.smithi006.stderr:==00:00:00:09.736 35451== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:20.247 INFO:tasks.ceph.osd.1.smithi006.stderr:==00:00:00:09.715 35454== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:20.343 INFO:tasks.ceph.osd.2.smithi006.stderr:==00:00:00:09.827 35452== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:20.348 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:09.829 35208== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:20.353 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:14:20.354 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:09.833 35210== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:20.385 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:09.866 35209== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:20.415 INFO:tasks.ceph.osd.3.smithi006.stderr:==00:00:00:09.884 35453== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:20.441 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.441+0000 7f231036d640 10 monclient: get_monmap_and_config 2024-03-25T19:14:20.441 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.441+0000 7f231036d640 10 monclient: build_initial_monmap 2024-03-25T19:14:20.441 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.441+0000 7f231036d640 10 monclient: monmap: 2024-03-25T19:14:20.442 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:14:20.442 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:20.442 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:14:20.442768+0000 2024-03-25T19:14:20.442 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:14:20.442768+0000 2024-03-25T19:14:20.442 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:14:20.442 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:20.442 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:14:20.442 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:14:20.442 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:14:20.442 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:20.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.442+0000 7f231036d640 1 Processor -- start 2024-03-25T19:14:20.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.442+0000 7f231036d640 1 -- start start 2024-03-25T19:14:20.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.442+0000 7f231036d640 10 monclient: init 2024-03-25T19:14:20.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.443+0000 7f231036d640 10 monclient: _reopen_session rank -1 2024-03-25T19:14:20.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.443+0000 7f231036d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:14:20.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.443+0000 7f231036d640 10 monclient(hunting): picked mon.noname-c con 0x7f230806ae10 addr v1:172.21.15.192:6789/0 2024-03-25T19:14:20.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.443+0000 7f231036d640 10 monclient(hunting): picked mon.noname-b con 0x7f23080723b0 addr v1:172.21.15.6:6790/0 2024-03-25T19:14:20.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.443+0000 7f231036d640 10 monclient(hunting): picked mon.noname-a con 0x7f23080aff70 addr v1:172.21.15.6:6789/0 2024-03-25T19:14:20.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.443+0000 7f231036d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2308142a70 con 0x7f23080aff70 2024-03-25T19:14:20.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.443+0000 7f231036d640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23080b1190 con 0x7f230806ae10 2024-03-25T19:14:20.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.443+0000 7f231036d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23080afa50 con 0x7f23080723b0 2024-03-25T19:14:20.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.443+0000 7f231036d640 10 monclient(hunting): _renew_subs 2024-03-25T19:14:20.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.443+0000 7f231036d640 10 monclient(hunting): authenticate will time out at 5964.682918s 2024-03-25T19:14:20.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.443+0000 7f230d908640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f23080723b0 0x7f230813f6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:58816/0 (socket says 172.21.15.6:58816) 2024-03-25T19:14:20.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.443+0000 7f230e90a640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f23080aff70 0x7f230814dc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:54706/0 (socket says 172.21.15.6:54706) 2024-03-25T19:14:20.445 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.443+0000 7f230d908640 1 -- 172.21.15.6:0/613290042 learned_addr learned my addr 172.21.15.6:0/613290042 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:14:20.446 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.446+0000 7f230d107640 1 -- 172.21.15.6:0/613290042 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1103248169 0 0) 0x7f2308142a70 con 0x7f23080aff70 2024-03-25T19:14:20.446 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.446+0000 7f230d107640 10 monclient(hunting): _negotiate 2024-03-25T19:14:20.446 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.446+0000 7f230d107640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:20.446 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.446+0000 7f230d107640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:20.446 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.446+0000 7f230d107640 10 monclient(hunting): my global_id is 4206 2024-03-25T19:14:20.446 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.446+0000 7f230d107640 1 -- 172.21.15.6:0/613290042 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22f4003520 con 0x7f23080aff70 2024-03-25T19:14:20.446 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.446+0000 7f230d107640 1 -- 172.21.15.6:0/613290042 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2376508088 0 0) 0x7f23080b1190 con 0x7f230806ae10 2024-03-25T19:14:20.446 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.446+0000 7f230d107640 10 monclient(hunting): _negotiate 2024-03-25T19:14:20.446 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.446+0000 7f230d107640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:20.446 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.446+0000 7f230d107640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:20.447 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.446+0000 7f230d107640 10 monclient(hunting): my global_id is 4204 2024-03-25T19:14:20.447 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.446+0000 7f230d107640 1 -- 172.21.15.6:0/613290042 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2308142a70 con 0x7f230806ae10 2024-03-25T19:14:20.447 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.446+0000 7f230d107640 1 -- 172.21.15.6:0/613290042 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3786737178 0 0) 0x7f23080afa50 con 0x7f23080723b0 2024-03-25T19:14:20.447 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.446+0000 7f230d107640 10 monclient(hunting): _negotiate 2024-03-25T19:14:20.447 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.446+0000 7f230d107640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:20.447 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.446+0000 7f230d107640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:20.447 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.446+0000 7f230d107640 10 monclient(hunting): my global_id is 4205 2024-03-25T19:14:20.447 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.446+0000 7f230d107640 1 -- 172.21.15.6:0/613290042 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23080b1190 con 0x7f23080723b0 2024-03-25T19:14:20.449 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.449+0000 7f230d107640 1 -- 172.21.15.6:0/613290042 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4091634632 0 0) 0x7f2308142a70 con 0x7f230806ae10 2024-03-25T19:14:20.449 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.449+0000 7f230d107640 1 -- 172.21.15.6:0/613290042 >> v1:172.21.15.6:6790/0 conn(0x7f23080723b0 legacy=0x7f230813f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:20.449 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.449+0000 7f230d107640 1 -- 172.21.15.6:0/613290042 >> v1:172.21.15.6:6789/0 conn(0x7f23080aff70 legacy=0x7f230814dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:20.449 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.449+0000 7f230d107640 10 monclient: _finish_hunting 0 2024-03-25T19:14:20.449 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.449+0000 7f230d107640 1 monclient: found mon.noname-c 2024-03-25T19:14:20.449 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.449+0000 7f230d107640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:14:20.449 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.449+0000 7f230d107640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:14:20.449 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.449+0000 7f230d107640 1 -- 172.21.15.6:0/613290042 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23080af670 con 0x7f230806ae10 2024-03-25T19:14:20.449 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.449+0000 7f230d107640 10 monclient: _finish_auth 0 2024-03-25T19:14:20.450 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.449+0000 7f230d107640 10 monclient: _check_auth_tickets 2024-03-25T19:14:20.450 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.449+0000 7f230d107640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:14:20.450 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.449+0000 7f230d107640 1 -- 172.21.15.6:0/613290042 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f23040015f0 con 0x7f230806ae10 2024-03-25T19:14:20.450 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.449+0000 7f230d107640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:20.450 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.449+0000 7f230d107640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:14:20.450 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.449+0000 7f230d107640 10 monclient: dump: 2024-03-25T19:14:20.450 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:14:20.450 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:20.450 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:20.450 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:20.450 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:20.450 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:20.450 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:20.451 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:20.451 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:20.451 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:20.451 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.449+0000 7f231036d640 5 monclient: authenticate success, global_id 4204 2024-03-25T19:14:20.451 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.449+0000 7f231036d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:14:20.454 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.450+0000 7f230d107640 1 -- 172.21.15.6:0/613290042 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2304001750 con 0x7f230806ae10 2024-03-25T19:14:20.454 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.450+0000 7f230d107640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:14:20.454 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.450+0000 7f230d107640 1 -- 172.21.15.6:0/613290042 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f2304001b60 con 0x7f230806ae10 2024-03-25T19:14:20.454 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.450+0000 7f230d107640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:20.454 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.450+0000 7f230d107640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:14:20.454 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.450+0000 7f230d107640 10 monclient: dump: 2024-03-25T19:14:20.454 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:14:20.454 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:20.454 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:20.454 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:20.454 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:20.454 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:20.454 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:20.455 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:20.455 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:20.455 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:20.455 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.450+0000 7f231036d640 10 monclient: get_monmap_and_config success 2024-03-25T19:14:20.455 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.450+0000 7f231036d640 4 set_mon_vals no callback set 2024-03-25T19:14:20.455 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.451+0000 7f231036d640 10 monclient: shutdown 2024-03-25T19:14:20.455 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.451+0000 7f231036d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:14:20.455 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.451+0000 7f231036d640 1 -- 172.21.15.6:0/613290042 >> v1:172.21.15.192:6789/0 conn(0x7f230806ae10 legacy=0x7f2308142360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:20.455 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.451+0000 7f231036d640 1 -- 172.21.15.6:0/613290042 shutdown_connections 2024-03-25T19:14:20.455 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.451+0000 7f231036d640 1 -- 172.21.15.6:0/613290042 >> 172.21.15.6:0/613290042 conn(0x7f23080b4c70 msgr2=0x7f23080b5080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:14:20.455 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.451+0000 7f231036d640 1 -- 172.21.15.6:0/613290042 shutdown_connections 2024-03-25T19:14:20.455 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.451+0000 7f231036d640 1 -- 172.21.15.6:0/613290042 wait complete. 2024-03-25T19:14:20.455 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.452+0000 7f231036d640 10 monclient: build_initial_monmap 2024-03-25T19:14:20.456 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.452+0000 7f231036d640 10 monclient: monmap: 2024-03-25T19:14:20.456 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:14:20.456 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:14:20.456 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:14:20.456 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:14:20.456 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:14:20.456 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:14:20.456 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:14:20.456 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:14:20.456 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:14:20.456 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:20.456 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.452+0000 7f231036d640 1 librados: starting msgr at 2024-03-25T19:14:20.456 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.452+0000 7f231036d640 1 librados: starting objecter 2024-03-25T19:14:20.457 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.452+0000 7f231036d640 1 Processor -- start 2024-03-25T19:14:20.457 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.452+0000 7f231036d640 1 -- start start 2024-03-25T19:14:20.457 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.452+0000 7f231036d640 1 librados: setting wanted keys 2024-03-25T19:14:20.457 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.452+0000 7f231036d640 1 librados: calling monclient init 2024-03-25T19:14:20.457 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.452+0000 7f231036d640 10 monclient: init 2024-03-25T19:14:20.457 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.453+0000 7f231036d640 10 monclient: _reopen_session rank -1 2024-03-25T19:14:20.457 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.453+0000 7f231036d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:14:20.457 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.453+0000 7f231036d640 10 monclient(hunting): picked mon.noname-a con 0x7f230806ae10 addr v1:172.21.15.6:6789/0 2024-03-25T19:14:20.457 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.453+0000 7f231036d640 10 monclient(hunting): picked mon.noname-c con 0x7f23080723b0 addr v1:172.21.15.6:6790/0 2024-03-25T19:14:20.457 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.453+0000 7f231036d640 10 monclient(hunting): picked mon.noname-b con 0x7f23080aff70 addr v1:172.21.15.192:6789/0 2024-03-25T19:14:20.457 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.453+0000 7f231036d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f230806f140 con 0x7f230806ae10 2024-03-25T19:14:20.457 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.453+0000 7f231036d640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2308136200 con 0x7f23080aff70 2024-03-25T19:14:20.457 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.453+0000 7f231036d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2308136470 con 0x7f23080723b0 2024-03-25T19:14:20.457 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.453+0000 7f231036d640 10 monclient(hunting): _renew_subs 2024-03-25T19:14:20.457 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.453+0000 7f231036d640 10 monclient(hunting): authenticate will time out at 5964.692953s 2024-03-25T19:14:20.458 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.453+0000 7f230e109640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f230806ae10 0x7f230806f5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:54708/0 (socket says 172.21.15.6:54708) 2024-03-25T19:14:20.458 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.453+0000 7f230e109640 1 -- 172.21.15.6:0/442254831 learned_addr learned my addr 172.21.15.6:0/442254831 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:14:20.458 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.453+0000 7f22feffd640 10 client.?.objecter ms_handle_connect 0x7f230806ae10 2024-03-25T19:14:20.458 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.453+0000 7f22feffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:20.458 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.453+0000 7f22feffd640 10 client.?.objecter ms_handle_connect 0x7f23080723b0 2024-03-25T19:14:20.458 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.453+0000 7f22feffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:20.458 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.454+0000 7f22feffd640 10 client.?.objecter ms_handle_connect 0x7f23080aff70 2024-03-25T19:14:20.458 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.454+0000 7f22feffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:20.459 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.455+0000 7f22feffd640 1 -- 172.21.15.6:0/442254831 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3713669103 0 0) 0x7f2308136200 con 0x7f23080aff70 2024-03-25T19:14:20.459 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.455+0000 7f22feffd640 10 monclient(hunting): _negotiate 2024-03-25T19:14:20.459 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.455+0000 7f22feffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:20.459 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.455+0000 7f22feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:20.459 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.455+0000 7f22feffd640 10 monclient(hunting): my global_id is 4207 2024-03-25T19:14:20.459 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.455+0000 7f22feffd640 1 -- 172.21.15.6:0/442254831 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22ec004eb0 con 0x7f23080aff70 2024-03-25T19:14:20.459 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.455+0000 7f22feffd640 1 -- 172.21.15.6:0/442254831 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4093190548 0 0) 0x7f230806f140 con 0x7f230806ae10 2024-03-25T19:14:20.459 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.455+0000 7f22feffd640 10 monclient(hunting): _negotiate 2024-03-25T19:14:20.459 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.455+0000 7f22feffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:20.460 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.455+0000 7f22feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:20.460 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.455+0000 7f22feffd640 10 monclient(hunting): my global_id is 4209 2024-03-25T19:14:20.460 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.455+0000 7f22feffd640 1 -- 172.21.15.6:0/442254831 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2308136200 con 0x7f230806ae10 2024-03-25T19:14:20.460 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.455+0000 7f22feffd640 1 -- 172.21.15.6:0/442254831 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3313077709 0 0) 0x7f2308136470 con 0x7f23080723b0 2024-03-25T19:14:20.460 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.455+0000 7f22feffd640 10 monclient(hunting): _negotiate 2024-03-25T19:14:20.460 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.455+0000 7f22feffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:20.460 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.455+0000 7f22feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:20.460 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.455+0000 7f22feffd640 10 monclient(hunting): my global_id is 4208 2024-03-25T19:14:20.460 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.455+0000 7f22feffd640 1 -- 172.21.15.6:0/442254831 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f230806f140 con 0x7f23080723b0 2024-03-25T19:14:20.461 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.460+0000 7f22feffd640 1 -- 172.21.15.6:0/442254831 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2636421300 0 0) 0x7f22ec004eb0 con 0x7f23080aff70 2024-03-25T19:14:20.461 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.460+0000 7f22feffd640 1 -- 172.21.15.6:0/442254831 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2308136470 con 0x7f23080aff70 2024-03-25T19:14:20.461 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.460+0000 7f22feffd640 1 -- 172.21.15.6:0/442254831 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f2300001c10 con 0x7f23080aff70 2024-03-25T19:14:20.461 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.460+0000 7f22feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:20.461 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.460+0000 7f22feffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:14:20.461 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.460+0000 7f22feffd640 10 monclient(hunting): dump: 2024-03-25T19:14:20.461 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:14:20.462 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:20.462 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:20.462 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:20.462 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:20.462 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:20.462 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:20.462 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:20.462 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:20.462 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:20.462 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.460+0000 7f22feffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:14:20.462 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.460+0000 7f22feffd640 1 -- 172.21.15.6:0/442254831 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 818351733 0 0) 0x7f2308136200 con 0x7f230806ae10 2024-03-25T19:14:20.462 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.460+0000 7f22feffd640 1 -- 172.21.15.6:0/442254831 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22ec004eb0 con 0x7f230806ae10 2024-03-25T19:14:20.462 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.460+0000 7f22feffd640 1 -- 172.21.15.6:0/442254831 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3146691735 0 0) 0x7f230806f140 con 0x7f23080723b0 2024-03-25T19:14:20.463 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.460+0000 7f22feffd640 1 -- 172.21.15.6:0/442254831 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2308136200 con 0x7f23080723b0 2024-03-25T19:14:20.463 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.460+0000 7f22feffd640 1 -- 172.21.15.6:0/442254831 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f22f80018f0 con 0x7f23080723b0 2024-03-25T19:14:20.463 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.460+0000 7f22feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:20.463 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.460+0000 7f22feffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:14:20.463 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.460+0000 7f22feffd640 10 monclient(hunting): dump: 2024-03-25T19:14:20.463 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:14:20.463 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:20.463 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:20.463 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:20.463 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:20.463 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:20.463 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:20.463 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:20.463 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:20.463 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:20.464 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.460+0000 7f22feffd640 1 -- 172.21.15.6:0/442254831 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f2304001b40 con 0x7f230806ae10 2024-03-25T19:14:20.464 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.460+0000 7f22feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:20.464 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.460+0000 7f22feffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:14:20.464 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.460+0000 7f22feffd640 10 monclient(hunting): dump: 2024-03-25T19:14:20.464 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:14:20.464 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:20.464 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:20.464 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:20.464 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:20.464 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:20.464 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:20.464 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:20.464 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:20.464 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:20.465 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.462+0000 7f22feffd640 1 -- 172.21.15.6:0/442254831 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1887941229 0 0) 0x7f2308136470 con 0x7f23080aff70 2024-03-25T19:14:20.465 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.462+0000 7f22feffd640 1 -- 172.21.15.6:0/442254831 >> v1:172.21.15.6:6790/0 conn(0x7f23080723b0 legacy=0x7f230806fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:20.466 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.462+0000 7f22feffd640 1 -- 172.21.15.6:0/442254831 >> v1:172.21.15.6:6789/0 conn(0x7f230806ae10 legacy=0x7f230806f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:20.466 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.462+0000 7f22feffd640 10 monclient: _finish_hunting 0 2024-03-25T19:14:20.466 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.462+0000 7f22feffd640 1 monclient: found mon.b 2024-03-25T19:14:20.466 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.462+0000 7f22feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:14:20.466 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.462+0000 7f22feffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:20.466 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.462+0000 7f22feffd640 1 -- 172.21.15.6:0/442254831 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2308136750 con 0x7f23080aff70 2024-03-25T19:14:20.466 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.462+0000 7f22feffd640 10 monclient: _finish_auth 0 2024-03-25T19:14:20.466 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.462+0000 7f22feffd640 10 monclient: _check_auth_tickets 2024-03-25T19:14:20.466 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.462+0000 7f22feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:14:20.466 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.462+0000 7f231036d640 5 monclient: authenticate success, global_id 4207 2024-03-25T19:14:20.466 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.462+0000 7f231036d640 10 monclient: _renew_subs 2024-03-25T19:14:20.466 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.462+0000 7f231036d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:20.466 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.462+0000 7f231036d640 1 -- 172.21.15.6:0/442254831 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23081399d0 con 0x7f23080aff70 2024-03-25T19:14:20.466 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.462+0000 7f231036d640 10 client.4207.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:14:20.466 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.462+0000 7f231036d640 10 monclient: _renew_subs 2024-03-25T19:14:20.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.462+0000 7f231036d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:20.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.462+0000 7f231036d640 1 -- 172.21.15.6:0/442254831 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2308071bf0 con 0x7f23080aff70 2024-03-25T19:14:20.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.462+0000 7f231036d640 1 librados: init done 2024-03-25T19:14:20.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.463+0000 7f231036d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:20.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.463+0000 7f231036d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:20.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.463+0000 7f231036d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:20.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.463+0000 7f231036d640 1 -- 172.21.15.6:0/442254831 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2308136c50 con 0x7f23080aff70 2024-03-25T19:14:20.529 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:09.995 35211== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:20.537 INFO:tasks.ceph.osd.0.smithi006.stderr:==00:00:00:10.029 35451== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:20.542 INFO:tasks.ceph.osd.1.smithi006.stderr:==00:00:00:10.011 35454== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:20.608 INFO:tasks.ceph.osd.2.smithi006.stderr:==00:00:00:10.092 35452== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:20.612 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:10.093 35208== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:20.647 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:10.126 35210== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:20.667 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:10.149 35209== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:20.679 INFO:tasks.ceph.osd.3.smithi006.stderr:==00:00:00:10.148 35453== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:20.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.724+0000 7f22feffd640 1 -- 172.21.15.6:0/442254831 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23000020e0 con 0x7f23080aff70 2024-03-25T19:14:20.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.724+0000 7f22feffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:14:20.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.724+0000 7f22feffd640 1 -- 172.21.15.6:0/442254831 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f2300004d00 con 0x7f23080aff70 2024-03-25T19:14:20.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.724+0000 7f22feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:20.724 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.724+0000 7f22feffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:14:20.724 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.724+0000 7f22feffd640 10 monclient: dump: 2024-03-25T19:14:20.724 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:14:20.724 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:20.724 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:20.724 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:20.724 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:20.724 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:20.724 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:20.724 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:20.724 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:20.724 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:20.724 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.724+0000 7f230c906640 4 set_mon_vals no callback set 2024-03-25T19:14:20.725 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.724+0000 7f22feffd640 1 -- 172.21.15.6:0/442254831 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f230001ad50 con 0x7f23080aff70 2024-03-25T19:14:20.725 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.724+0000 7f230e109640 10 monclient: get_auth_request con 0x7f22ec06e070 auth_method 0 2024-03-25T19:14:20.725 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.725+0000 7f22feffd640 10 client.4207.objecter ms_handle_connect 0x7f22ec06e070 2024-03-25T19:14:20.727 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.727+0000 7f22feffd640 1 -- 172.21.15.6:0/442254831 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f2300054a40 con 0x7f23080aff70 2024-03-25T19:14:20.727 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.727+0000 7f22feffd640 10 client.4207.objecter ms_dispatch 0x7f2308000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:14:20.727 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.727+0000 7f22feffd640 3 client.4207.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:14:20.727 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.727+0000 7f22feffd640 3 client.4207.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:14:20.727 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.727+0000 7f22feffd640 20 client.4207.objecter dump_active .. 0 homeless 2024-03-25T19:14:20.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.728+0000 7f22feffd640 1 -- 172.21.15.6:0/442254831 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2300050380 con 0x7f23080aff70 2024-03-25T19:14:20.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.728+0000 7f22feffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:20.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.728+0000 7f22feffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:14:20.832 INFO:tasks.ceph.osd.0.smithi006.stderr:==00:00:00:10.324 35451== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:20.836 INFO:tasks.ceph.osd.1.smithi006.stderr:==00:00:00:10.304 35454== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:20.839 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:10.305 35211== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:20.879 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:10.360 35208== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:20.904 INFO:tasks.ceph.osd.2.smithi006.stderr:==00:00:00:10.388 35452== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:20.912 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.912+0000 7f231036d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:14:20.912 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.912+0000 7f231036d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:14:20.912 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.912+0000 7f231036d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:20.913 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.912+0000 7f231036d640 1 -- 172.21.15.6:0/442254831 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f230804f730 con 0x7f23080aff70 2024-03-25T19:14:20.937 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:10.418 35209== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:20.945 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:10.424 35210== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:20.945 INFO:tasks.ceph.osd.3.smithi006.stderr:==00:00:00:10.415 35453== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:20.955 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.956+0000 7f22feffd640 1 -- 172.21.15.6:0/442254831 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f2300052480 con 0x7f23080aff70 2024-03-25T19:14:20.956 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.956+0000 7f22feffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:14:20.956 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.956+0000 7f22feffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:14:20.956 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:14:20.956 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:14:20.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.957+0000 7f231036d640 10 librados: watch_flush enter 2024-03-25T19:14:20.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.957+0000 7f231036d640 10 librados: watch_flush exit 2024-03-25T19:14:20.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.957+0000 7f231036d640 20 client.4207.objecter shutdown clearing up homeless session... 2024-03-25T19:14:20.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.957+0000 7f231036d640 10 client.4207.objecter successfully canceled tick 2024-03-25T19:14:20.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.957+0000 7f231036d640 1 -- 172.21.15.6:0/442254831 >> v1:172.21.15.6:6800/2936652650 conn(0x7f22ec06e070 legacy=0x7f22ec070530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:20.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.957+0000 7f231036d640 10 monclient: shutdown 2024-03-25T19:14:20.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.957+0000 7f231036d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:14:20.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.957+0000 7f231036d640 1 -- 172.21.15.6:0/442254831 >> v1:172.21.15.192:6789/0 conn(0x7f23080aff70 legacy=0x7f23081e1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:20.959 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.957+0000 7f231036d640 1 -- 172.21.15.6:0/442254831 shutdown_connections 2024-03-25T19:14:20.959 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.957+0000 7f231036d640 1 -- 172.21.15.6:0/442254831 >> 172.21.15.6:0/442254831 conn(0x7f23080b4c70 msgr2=0x7f230813f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:14:20.959 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.958+0000 7f231036d640 1 -- 172.21.15.6:0/442254831 shutdown_connections 2024-03-25T19:14:20.959 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.958+0000 7f231036d640 1 -- 172.21.15.6:0/442254831 wait complete. 2024-03-25T19:14:20.959 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:20.958+0000 7f231036d640 1 librados: shutdown 2024-03-25T19:14:20.967 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:14:21.128 INFO:tasks.ceph.osd.0.smithi006.stderr:==00:00:00:10.620 35451== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:21.130 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:10.596 35211== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:21.131 INFO:tasks.ceph.osd.1.smithi006.stderr:==00:00:00:10.600 35454== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:21.197 INFO:tasks.ceph.osd.2.smithi006.stderr:==00:00:00:10.681 35452== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:21.218 INFO:tasks.ceph.osd.3.smithi006.stderr:==00:00:00:10.687 35453== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:21.237 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:10.717 35210== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:21.365 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:10.846 35208== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:21.416 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:10.897 35209== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:21.428 INFO:tasks.ceph.osd.0.smithi006.stderr:==00:00:00:10.920 35451== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:21.429 INFO:tasks.ceph.osd.1.smithi006.stderr:==00:00:00:10.898 35454== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:21.436 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:10.918 35208== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:21.471 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:10.952 35209== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:21.490 INFO:tasks.ceph.osd.3.smithi006.stderr:==00:00:00:10.960 35453== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:21.495 INFO:tasks.ceph.osd.2.smithi006.stderr:==00:00:00:10.980 35452== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:21.547 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:11.026 35210== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:21.604 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:11.083 35210== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:21.663 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:11.129 35211== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:21.718 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:11.184 35211== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:21.741 INFO:tasks.ceph.osd.0.smithi006.stderr:==00:00:00:11.233 35451== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:21.760 INFO:tasks.ceph.osd.3.smithi006.stderr:==00:00:00:11.229 35453== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:21.982 INFO:tasks.ceph.osd.1.smithi006.stderr:==00:00:00:11.451 35454== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:22.027 INFO:tasks.ceph.osd.2.smithi006.stderr:==00:00:00:11.511 35452== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:22.038 INFO:tasks.ceph.osd.1.smithi006.stderr:==00:00:00:11.506 35454== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:22.084 INFO:tasks.ceph.osd.2.smithi006.stderr:==00:00:00:11.568 35452== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:22.252 INFO:tasks.ceph.osd.3.smithi006.stderr:==00:00:00:11.721 35453== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:22.273 INFO:tasks.ceph.osd.0.smithi006.stderr:==00:00:00:11.765 35451== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:22.322 INFO:tasks.ceph.osd.3.smithi006.stderr:==00:00:00:11.791 35453== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:22.329 INFO:tasks.ceph.osd.0.smithi006.stderr:==00:00:00:11.822 35451== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:26.732 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:16.213 35209== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:26.764 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:16.245 35208== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:26.851 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:16.317 35211== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:27.000 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:16.480 35210== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:27.762 INFO:tasks.ceph.osd.2.smithi006.stderr:==00:00:00:17.246 35452== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:27.772 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:14:27.866 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.866+0000 7fba11cfb640 10 monclient: get_monmap_and_config 2024-03-25T19:14:27.866 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.866+0000 7fba11cfb640 10 monclient: build_initial_monmap 2024-03-25T19:14:27.867 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.866+0000 7fba11cfb640 10 monclient: monmap: 2024-03-25T19:14:27.867 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:14:27.867 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:27.867 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:14:27.867725+0000 2024-03-25T19:14:27.867 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:14:27.867725+0000 2024-03-25T19:14:27.867 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:14:27.867 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:27.867 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:14:27.867 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:14:27.867 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:14:27.867 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:27.868 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.867+0000 7fba11cfb640 1 Processor -- start 2024-03-25T19:14:27.868 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.867+0000 7fba11cfb640 1 -- start start 2024-03-25T19:14:27.869 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.867+0000 7fba11cfb640 10 monclient: init 2024-03-25T19:14:27.869 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.868+0000 7fba11cfb640 10 monclient: _reopen_session rank -1 2024-03-25T19:14:27.869 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.868+0000 7fba11cfb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:14:27.869 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.868+0000 7fba11cfb640 10 monclient(hunting): picked mon.noname-a con 0x7fba0c0b4e70 addr v1:172.21.15.6:6789/0 2024-03-25T19:14:27.869 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.868+0000 7fba11cfb640 10 monclient(hunting): picked mon.noname-c con 0x7fba0c0b44b0 addr v1:172.21.15.192:6789/0 2024-03-25T19:14:27.869 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.868+0000 7fba11cfb640 10 monclient(hunting): picked mon.noname-b con 0x7fba0c0b1b00 addr v1:172.21.15.6:6790/0 2024-03-25T19:14:27.869 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.868+0000 7fba11cfb640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba0c0a96f0 con 0x7fba0c0b4e70 2024-03-25T19:14:27.869 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.868+0000 7fba11cfb640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba0c0ab0e0 con 0x7fba0c0b44b0 2024-03-25T19:14:27.869 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.868+0000 7fba11cfb640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba0c145f70 con 0x7fba0c0b1b00 2024-03-25T19:14:27.869 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.868+0000 7fba11cfb640 10 monclient(hunting): _renew_subs 2024-03-25T19:14:27.869 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.868+0000 7fba11cfb640 10 monclient(hunting): authenticate will time out at 5972.107969s 2024-03-25T19:14:27.869 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.868+0000 7fba0bfff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fba0c0b1b00 0x7fba0c14dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:58824/0 (socket says 172.21.15.6:58824) 2024-03-25T19:14:27.869 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.868+0000 7fba0bfff640 1 -- 172.21.15.6:0/2931466285 learned_addr learned my addr 172.21.15.6:0/2931466285 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:14:27.872 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.871+0000 7fba0a7fc640 1 -- 172.21.15.6:0/2931466285 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3782075090 0 0) 0x7fba0c145f70 con 0x7fba0c0b1b00 2024-03-25T19:14:27.872 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.871+0000 7fba0a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:14:27.872 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.871+0000 7fba0a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:27.872 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.871+0000 7fba0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:27.872 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.871+0000 7fba0a7fc640 10 monclient(hunting): my global_id is 4211 2024-03-25T19:14:27.872 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.871+0000 7fba0a7fc640 1 -- 172.21.15.6:0/2931466285 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9f4003520 con 0x7fba0c0b1b00 2024-03-25T19:14:27.872 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.871+0000 7fba0a7fc640 1 -- 172.21.15.6:0/2931466285 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4045949570 0 0) 0x7fba0c0a96f0 con 0x7fba0c0b4e70 2024-03-25T19:14:27.872 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.871+0000 7fba0a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:14:27.872 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.871+0000 7fba0a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:27.872 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.871+0000 7fba0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:27.872 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.871+0000 7fba0a7fc640 10 monclient(hunting): my global_id is 4212 2024-03-25T19:14:27.872 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.871+0000 7fba0a7fc640 1 -- 172.21.15.6:0/2931466285 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba0c145f70 con 0x7fba0c0b4e70 2024-03-25T19:14:27.872 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.871+0000 7fba0a7fc640 1 -- 172.21.15.6:0/2931466285 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 950396568 0 0) 0x7fba0c0ab0e0 con 0x7fba0c0b44b0 2024-03-25T19:14:27.872 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.871+0000 7fba0a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:14:27.872 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.871+0000 7fba0a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:27.873 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.871+0000 7fba0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:27.873 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.871+0000 7fba0a7fc640 10 monclient(hunting): my global_id is 4210 2024-03-25T19:14:27.873 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.871+0000 7fba0a7fc640 1 -- 172.21.15.6:0/2931466285 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba0c0a96f0 con 0x7fba0c0b44b0 2024-03-25T19:14:27.876 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.876+0000 7fba0a7fc640 1 -- 172.21.15.6:0/2931466285 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4002577669 0 0) 0x7fb9f4003520 con 0x7fba0c0b1b00 2024-03-25T19:14:27.876 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.876+0000 7fba0a7fc640 1 -- 172.21.15.6:0/2931466285 >> v1:172.21.15.192:6789/0 conn(0x7fba0c0b44b0 legacy=0x7fba0c145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:27.876 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.876+0000 7fba0a7fc640 1 -- 172.21.15.6:0/2931466285 >> v1:172.21.15.6:6789/0 conn(0x7fba0c0b4e70 legacy=0x7fba0c13f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:27.876 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.876+0000 7fba0a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:14:27.876 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.876+0000 7fba0a7fc640 1 monclient: found mon.noname-b 2024-03-25T19:14:27.876 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.876+0000 7fba0a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:14:27.876 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.876+0000 7fba0a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-03-25T19:14:27.876 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.876+0000 7fba0a7fc640 1 -- 172.21.15.6:0/2931466285 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba0c0a9b80 con 0x7fba0c0b1b00 2024-03-25T19:14:27.876 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.876+0000 7fba0a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:14:27.877 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.876+0000 7fba0a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:14:27.877 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.876+0000 7fba0a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:14:27.877 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.876+0000 7fba0a7fc640 1 -- 172.21.15.6:0/2931466285 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fba04001a10 con 0x7fba0c0b1b00 2024-03-25T19:14:27.877 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.876+0000 7fba0a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:27.877 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.876+0000 7fba0a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:14:27.877 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.876+0000 7fba0a7fc640 10 monclient: dump: 2024-03-25T19:14:27.877 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:14:27.877 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:27.877 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:27.877 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:27.877 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:27.877 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:27.877 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:27.877 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:27.877 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:27.877 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:27.877 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.876+0000 7fba11cfb640 5 monclient: authenticate success, global_id 4211 2024-03-25T19:14:27.878 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.876+0000 7fba11cfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:14:27.878 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.878+0000 7fba0a7fc640 1 -- 172.21.15.6:0/2931466285 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba04001b70 con 0x7fba0c0b1b00 2024-03-25T19:14:27.878 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.878+0000 7fba0a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:14:27.878 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.878+0000 7fba0a7fc640 1 -- 172.21.15.6:0/2931466285 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fba04001f80 con 0x7fba0c0b1b00 2024-03-25T19:14:27.878 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.878+0000 7fba0a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:27.878 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.878+0000 7fba0a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:14:27.879 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.878+0000 7fba0a7fc640 10 monclient: dump: 2024-03-25T19:14:27.879 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:14:27.879 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:27.879 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:27.879 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:27.879 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:27.879 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:27.879 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:27.879 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:27.879 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:27.879 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:27.879 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.878+0000 7fba11cfb640 10 monclient: get_monmap_and_config success 2024-03-25T19:14:27.879 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.878+0000 7fba11cfb640 4 set_mon_vals no callback set 2024-03-25T19:14:27.881 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.879+0000 7fba11cfb640 10 monclient: shutdown 2024-03-25T19:14:27.881 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.879+0000 7fba11cfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:14:27.881 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.879+0000 7fba11cfb640 1 -- 172.21.15.6:0/2931466285 >> v1:172.21.15.6:6790/0 conn(0x7fba0c0b1b00 legacy=0x7fba0c14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:27.881 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.879+0000 7fba11cfb640 1 -- 172.21.15.6:0/2931466285 shutdown_connections 2024-03-25T19:14:27.881 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.879+0000 7fba11cfb640 1 -- 172.21.15.6:0/2931466285 >> 172.21.15.6:0/2931466285 conn(0x7fba0c0617a0 msgr2=0x7fba0c061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:14:27.881 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.879+0000 7fba11cfb640 1 -- 172.21.15.6:0/2931466285 shutdown_connections 2024-03-25T19:14:27.881 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.879+0000 7fba11cfb640 1 -- 172.21.15.6:0/2931466285 wait complete. 2024-03-25T19:14:27.881 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.880+0000 7fba11cfb640 10 monclient: build_initial_monmap 2024-03-25T19:14:27.881 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.880+0000 7fba11cfb640 10 monclient: monmap: 2024-03-25T19:14:27.881 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:14:27.882 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:14:27.882 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:14:27.882 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:14:27.882 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:14:27.882 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:14:27.882 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:14:27.882 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:14:27.882 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:14:27.882 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:27.882 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.880+0000 7fba11cfb640 1 librados: starting msgr at 2024-03-25T19:14:27.882 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.880+0000 7fba11cfb640 1 librados: starting objecter 2024-03-25T19:14:27.882 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.880+0000 7fba11cfb640 1 Processor -- start 2024-03-25T19:14:27.882 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.880+0000 7fba11cfb640 1 -- start start 2024-03-25T19:14:27.882 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.880+0000 7fba11cfb640 1 librados: setting wanted keys 2024-03-25T19:14:27.882 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.880+0000 7fba11cfb640 1 librados: calling monclient init 2024-03-25T19:14:27.883 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.880+0000 7fba11cfb640 10 monclient: init 2024-03-25T19:14:27.883 INFO:tasks.ceph.osd.3.smithi006.stderr:==00:00:00:17.350 35453== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:27.884 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.881+0000 7fba11cfb640 10 monclient: _reopen_session rank -1 2024-03-25T19:14:27.884 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.881+0000 7fba11cfb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:14:27.884 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.881+0000 7fba11cfb640 10 monclient(hunting): picked mon.noname-a con 0x7fba0c0b1b00 addr v1:172.21.15.6:6789/0 2024-03-25T19:14:27.884 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.881+0000 7fba11cfb640 10 monclient(hunting): picked mon.noname-b con 0x7fba0c0b44b0 addr v1:172.21.15.192:6789/0 2024-03-25T19:14:27.884 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.881+0000 7fba11cfb640 10 monclient(hunting): picked mon.noname-c con 0x7fba0c0b4e70 addr v1:172.21.15.6:6790/0 2024-03-25T19:14:27.884 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.881+0000 7fba11cfb640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba0c143280 con 0x7fba0c0b1b00 2024-03-25T19:14:27.884 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.881+0000 7fba11cfb640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba0c1d7a70 con 0x7fba0c0b44b0 2024-03-25T19:14:27.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.881+0000 7fba11cfb640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba0c1d4540 con 0x7fba0c0b4e70 2024-03-25T19:14:27.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.881+0000 7fba11cfb640 10 monclient(hunting): _renew_subs 2024-03-25T19:14:27.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.881+0000 7fba11cfb640 10 monclient(hunting): authenticate will time out at 5972.121297s 2024-03-25T19:14:27.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.882+0000 7fba0bfff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fba0c0b4e70 0x7fba0c1d4aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:58836/0 (socket says 172.21.15.6:58836) 2024-03-25T19:14:27.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.882+0000 7fba0bfff640 1 -- 172.21.15.6:0/2679351290 learned_addr learned my addr 172.21.15.6:0/2679351290 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:14:27.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.882+0000 7fb9f3fff640 10 client.?.objecter ms_handle_connect 0x7fba0c0b4e70 2024-03-25T19:14:27.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.882+0000 7fb9f3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:27.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.882+0000 7fb9f3fff640 10 client.?.objecter ms_handle_connect 0x7fba0c0b44b0 2024-03-25T19:14:27.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.882+0000 7fb9f3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:27.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.882+0000 7fb9f3fff640 10 client.?.objecter ms_handle_connect 0x7fba0c0b1b00 2024-03-25T19:14:27.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.882+0000 7fb9f3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:27.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.884+0000 7fb9f3fff640 1 -- 172.21.15.6:0/2679351290 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1001464886 0 0) 0x7fba0c1d7a70 con 0x7fba0c0b44b0 2024-03-25T19:14:27.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.884+0000 7fb9f3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:14:27.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.884+0000 7fb9f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:27.886 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.884+0000 7fb9f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:27.886 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.884+0000 7fb9f3fff640 10 monclient(hunting): my global_id is 4213 2024-03-25T19:14:27.886 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.884+0000 7fb9f3fff640 1 -- 172.21.15.6:0/2679351290 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9f8004eb0 con 0x7fba0c0b44b0 2024-03-25T19:14:27.887 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.885+0000 7fb9f3fff640 1 -- 172.21.15.6:0/2679351290 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 635851579 0 0) 0x7fba0c1d4540 con 0x7fba0c0b4e70 2024-03-25T19:14:27.887 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.885+0000 7fb9f3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:14:27.887 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.885+0000 7fb9f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:27.887 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.885+0000 7fb9f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:27.887 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.885+0000 7fb9f3fff640 10 monclient(hunting): my global_id is 4214 2024-03-25T19:14:27.887 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.885+0000 7fb9f3fff640 1 -- 172.21.15.6:0/2679351290 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba0c1d7a70 con 0x7fba0c0b4e70 2024-03-25T19:14:27.887 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.885+0000 7fb9f3fff640 1 -- 172.21.15.6:0/2679351290 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3144635221 0 0) 0x7fba0c143280 con 0x7fba0c0b1b00 2024-03-25T19:14:27.887 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.885+0000 7fb9f3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:14:27.887 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.885+0000 7fb9f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:27.887 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.885+0000 7fb9f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:27.887 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.885+0000 7fb9f3fff640 10 monclient(hunting): my global_id is 4215 2024-03-25T19:14:27.887 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.885+0000 7fb9f3fff640 1 -- 172.21.15.6:0/2679351290 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba0c1d4540 con 0x7fba0c0b1b00 2024-03-25T19:14:27.889 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.888+0000 7fb9f3fff640 1 -- 172.21.15.6:0/2679351290 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 213920311 0 0) 0x7fb9f8004eb0 con 0x7fba0c0b44b0 2024-03-25T19:14:27.889 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.888+0000 7fb9f3fff640 1 -- 172.21.15.6:0/2679351290 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba0c143280 con 0x7fba0c0b44b0 2024-03-25T19:14:27.889 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.888+0000 7fb9f3fff640 1 -- 172.21.15.6:0/2679351290 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fba000018f0 con 0x7fba0c0b44b0 2024-03-25T19:14:27.889 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.888+0000 7fb9f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:27.889 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.888+0000 7fb9f3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:14:27.889 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.888+0000 7fb9f3fff640 10 monclient(hunting): dump: 2024-03-25T19:14:27.889 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:14:27.889 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:27.889 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:27.889 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:27.890 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:27.890 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:27.890 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:27.890 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:27.890 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:27.890 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:27.890 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.888+0000 7fb9f3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:14:27.894 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.892+0000 7fb9f3fff640 1 -- 172.21.15.6:0/2679351290 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 69937879 0 0) 0x7fba0c143280 con 0x7fba0c0b44b0 2024-03-25T19:14:27.894 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.892+0000 7fb9f3fff640 1 -- 172.21.15.6:0/2679351290 >> v1:172.21.15.6:6790/0 conn(0x7fba0c0b4e70 legacy=0x7fba0c1d4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:27.894 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.892+0000 7fb9f3fff640 1 -- 172.21.15.6:0/2679351290 >> v1:172.21.15.6:6789/0 conn(0x7fba0c0b1b00 legacy=0x7fba0c141ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:27.894 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.892+0000 7fb9f3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:14:27.894 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.892+0000 7fb9f3fff640 1 monclient: found mon.b 2024-03-25T19:14:27.894 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.892+0000 7fb9f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:14:27.894 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.892+0000 7fb9f3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:27.894 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.892+0000 7fb9f3fff640 1 -- 172.21.15.6:0/2679351290 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba0c1d46d0 con 0x7fba0c0b44b0 2024-03-25T19:14:27.895 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.892+0000 7fb9f3fff640 10 monclient: _finish_auth 0 2024-03-25T19:14:27.895 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.892+0000 7fb9f3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:14:27.895 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.892+0000 7fb9f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:14:27.895 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.892+0000 7fba11cfb640 5 monclient: authenticate success, global_id 4213 2024-03-25T19:14:27.895 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.892+0000 7fba11cfb640 10 monclient: _renew_subs 2024-03-25T19:14:27.895 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.892+0000 7fba11cfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:27.895 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.892+0000 7fba11cfb640 1 -- 172.21.15.6:0/2679351290 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba0c1423b0 con 0x7fba0c0b44b0 2024-03-25T19:14:27.895 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.893+0000 7fba11cfb640 10 client.4213.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:14:27.895 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.893+0000 7fba11cfb640 10 monclient: _renew_subs 2024-03-25T19:14:27.895 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.893+0000 7fba11cfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:27.895 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.893+0000 7fba11cfb640 1 -- 172.21.15.6:0/2679351290 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fba0c1d8880 con 0x7fba0c0b44b0 2024-03-25T19:14:27.895 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.893+0000 7fba11cfb640 1 librados: init done 2024-03-25T19:14:27.896 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.895+0000 7fba11cfb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:27.896 INFO:tasks.ceph.osd.0.smithi006.stderr:==00:00:00:17.387 35451== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:27.896 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.895+0000 7fba11cfb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:27.896 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.895+0000 7fba11cfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:27.897 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.895+0000 7fba11cfb640 1 -- 172.21.15.6:0/2679351290 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba0c04f730 con 0x7fba0c0b44b0 2024-03-25T19:14:27.908 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.908+0000 7fb9f3fff640 1 -- 172.21.15.6:0/2679351290 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba00003610 con 0x7fba0c0b44b0 2024-03-25T19:14:27.908 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.908+0000 7fb9f3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:14:27.909 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.908+0000 7fba09ffb640 4 set_mon_vals no callback set 2024-03-25T19:14:27.909 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.909+0000 7fb9f3fff640 1 -- 172.21.15.6:0/2679351290 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fba00003880 con 0x7fba0c0b44b0 2024-03-25T19:14:27.909 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.909+0000 7fb9f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:27.909 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.909+0000 7fb9f3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:14:27.910 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.909+0000 7fb9f3fff640 10 monclient: dump: 2024-03-25T19:14:27.910 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:14:27.910 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:27.910 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:27.910 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:27.910 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:27.910 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:27.910 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:27.910 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:27.910 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:27.910 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:27.910 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.911+0000 7fb9f3fff640 1 -- 172.21.15.6:0/2679351290 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7fba000039e0 con 0x7fba0c0b44b0 2024-03-25T19:14:27.911 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.911+0000 7fb9f3fff640 1 -- 172.21.15.6:0/2679351290 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7fba00053290 con 0x7fba0c0b44b0 2024-03-25T19:14:27.911 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.911+0000 7fb9f3fff640 10 client.4213.objecter ms_dispatch 0x7fba0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:14:27.911 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.911+0000 7fb9f3fff640 3 client.4213.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:14:27.911 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.911+0000 7fb9f3fff640 3 client.4213.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:14:27.912 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.911+0000 7fb9f3fff640 20 client.4213.objecter dump_active .. 0 homeless 2024-03-25T19:14:27.914 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.914+0000 7fba0b7fe640 10 monclient: get_auth_request con 0x7fb9f806dfe0 auth_method 0 2024-03-25T19:14:27.914 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:27.914+0000 7fb9f3fff640 10 client.4213.objecter ms_handle_connect 0x7fb9f806dfe0 2024-03-25T19:14:28.086 INFO:tasks.ceph.osd.1.smithi006.stderr:==00:00:00:17.555 35454== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:28.175 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:28.175+0000 7fb9f3fff640 1 -- 172.21.15.6:0/2679351290 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fba0004f530 con 0x7fba0c0b44b0 2024-03-25T19:14:28.175 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:28.175+0000 7fb9f3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:28.175 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:28.175+0000 7fb9f3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:14:28.385 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:28.385+0000 7fba11cfb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:14:28.386 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:28.385+0000 7fba11cfb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:14:28.386 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:28.385+0000 7fba11cfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:28.386 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:28.385+0000 7fba11cfb640 1 -- 172.21.15.6:0/2679351290 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fba0c1d46d0 con 0x7fba0c0b44b0 2024-03-25T19:14:28.401 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:28.401+0000 7fb9f3fff640 1 -- 172.21.15.6:0/2679351290 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7fba0c1d46d0 con 0x7fba0c0b44b0 2024-03-25T19:14:28.401 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:28.401+0000 7fb9f3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:14:28.401 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:28.401+0000 7fb9f3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:14:28.401 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:14:28.401 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:14:28.403 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:28.402+0000 7fba11cfb640 10 librados: watch_flush enter 2024-03-25T19:14:28.403 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:28.403+0000 7fba11cfb640 10 librados: watch_flush exit 2024-03-25T19:14:28.403 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:28.403+0000 7fba11cfb640 20 client.4213.objecter shutdown clearing up homeless session... 2024-03-25T19:14:28.403 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:28.403+0000 7fba11cfb640 10 client.4213.objecter successfully canceled tick 2024-03-25T19:14:28.403 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:28.403+0000 7fba11cfb640 1 -- 172.21.15.6:0/2679351290 >> v1:172.21.15.6:6800/2936652650 conn(0x7fb9f806dfe0 legacy=0x7fb9f80704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:28.403 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:28.403+0000 7fba11cfb640 10 monclient: shutdown 2024-03-25T19:14:28.404 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:28.403+0000 7fba11cfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:14:28.404 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:28.403+0000 7fba11cfb640 1 -- 172.21.15.6:0/2679351290 >> v1:172.21.15.192:6789/0 conn(0x7fba0c0b44b0 legacy=0x7fba0c1d7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:28.404 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:28.403+0000 7fba11cfb640 1 -- 172.21.15.6:0/2679351290 shutdown_connections 2024-03-25T19:14:28.404 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:28.403+0000 7fba11cfb640 1 -- 172.21.15.6:0/2679351290 >> 172.21.15.6:0/2679351290 conn(0x7fba0c0617a0 msgr2=0x7fba0c144d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:14:28.404 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:28.403+0000 7fba11cfb640 1 -- 172.21.15.6:0/2679351290 shutdown_connections 2024-03-25T19:14:28.404 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:28.403+0000 7fba11cfb640 1 -- 172.21.15.6:0/2679351290 wait complete. 2024-03-25T19:14:28.404 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:28.403+0000 7fba11cfb640 1 librados: shutdown 2024-03-25T19:14:28.413 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:14:32.574 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:22.055 35209== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:32.577 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:22.059 35209== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:32.750 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:22.216 35211== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:32.754 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:22.220 35211== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:32.758 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:22.239 35208== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:32.762 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:22.243 35208== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:32.998 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:22.478 35210== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:33.002 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:22.482 35210== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:33.440 INFO:tasks.ceph.osd.0.smithi006.stderr:==00:00:00:22.933 35451== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:33.445 INFO:tasks.ceph.osd.0.smithi006.stderr:==00:00:00:22.938 35451== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:33.656 INFO:tasks.ceph.osd.2.smithi006.stderr:==00:00:00:23.140 35452== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:33.661 INFO:tasks.ceph.osd.2.smithi006.stderr:==00:00:00:23.145 35452== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:33.679 INFO:tasks.ceph.osd.3.smithi006.stderr:==00:00:00:23.148 35453== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:33.683 INFO:tasks.ceph.osd.3.smithi006.stderr:==00:00:00:23.152 35453== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:33.976 INFO:tasks.ceph.osd.1.smithi006.stderr:==00:00:00:23.445 35454== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:33.982 INFO:tasks.ceph.osd.1.smithi006.stderr:==00:00:00:23.451 35454== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:35.219 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:14:35.354 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.354+0000 7f577b1e9640 10 monclient: get_monmap_and_config 2024-03-25T19:14:35.355 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.354+0000 7f577b1e9640 10 monclient: build_initial_monmap 2024-03-25T19:14:35.355 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.354+0000 7f577b1e9640 10 monclient: monmap: 2024-03-25T19:14:35.355 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:14:35.355 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:35.355 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:14:35.355769+0000 2024-03-25T19:14:35.355 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:14:35.355769+0000 2024-03-25T19:14:35.355 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:14:35.355 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:35.355 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:14:35.355 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:14:35.355 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:14:35.355 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:35.357 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.355+0000 7f577b1e9640 1 Processor -- start 2024-03-25T19:14:35.357 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.355+0000 7f577b1e9640 1 -- start start 2024-03-25T19:14:35.357 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.355+0000 7f577b1e9640 10 monclient: init 2024-03-25T19:14:35.357 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.356+0000 7f577b1e9640 10 monclient: _reopen_session rank -1 2024-03-25T19:14:35.357 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.356+0000 7f577b1e9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:14:35.357 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.356+0000 7f577b1e9640 10 monclient(hunting): picked mon.noname-b con 0x7f57740b4e70 addr v1:172.21.15.6:6790/0 2024-03-25T19:14:35.357 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.356+0000 7f577b1e9640 10 monclient(hunting): picked mon.noname-c con 0x7f57740b44b0 addr v1:172.21.15.192:6789/0 2024-03-25T19:14:35.357 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.356+0000 7f577b1e9640 10 monclient(hunting): picked mon.noname-a con 0x7f57740b1b00 addr v1:172.21.15.6:6789/0 2024-03-25T19:14:35.357 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.356+0000 7f577b1e9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57740a96f0 con 0x7f57740b1b00 2024-03-25T19:14:35.358 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.356+0000 7f577b1e9640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57740ab0e0 con 0x7f57740b44b0 2024-03-25T19:14:35.358 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.356+0000 7f577b1e9640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5774145f70 con 0x7f57740b4e70 2024-03-25T19:14:35.358 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.357+0000 7f577b1e9640 10 monclient(hunting): _renew_subs 2024-03-25T19:14:35.358 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.357+0000 7f577b1e9640 10 monclient(hunting): authenticate will time out at 5979.596785s 2024-03-25T19:14:35.358 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.357+0000 7f5779786640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f57740b1b00 0x7f577414dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45472/0 (socket says 172.21.15.6:45472) 2024-03-25T19:14:35.358 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.357+0000 7f5779786640 1 -- 172.21.15.6:0/4186497808 learned_addr learned my addr 172.21.15.6:0/4186497808 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:14:35.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.360+0000 7f576b7fe640 1 -- 172.21.15.6:0/4186497808 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 596523703 0 0) 0x7f5774145f70 con 0x7f57740b4e70 2024-03-25T19:14:35.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.360+0000 7f576b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:14:35.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.360+0000 7f576b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:35.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.360+0000 7f576b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:35.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.360+0000 7f576b7fe640 10 monclient(hunting): my global_id is 4217 2024-03-25T19:14:35.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.360+0000 7f576b7fe640 1 -- 172.21.15.6:0/4186497808 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f575c003520 con 0x7f57740b4e70 2024-03-25T19:14:35.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.360+0000 7f576b7fe640 1 -- 172.21.15.6:0/4186497808 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4246216380 0 0) 0x7f57740ab0e0 con 0x7f57740b44b0 2024-03-25T19:14:35.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.360+0000 7f576b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:14:35.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.360+0000 7f576b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:35.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.360+0000 7f576b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:35.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.360+0000 7f576b7fe640 10 monclient(hunting): my global_id is 4216 2024-03-25T19:14:35.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.360+0000 7f576b7fe640 1 -- 172.21.15.6:0/4186497808 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5774145f70 con 0x7f57740b44b0 2024-03-25T19:14:35.363 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.361+0000 7f576b7fe640 1 -- 172.21.15.6:0/4186497808 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 565978527 0 0) 0x7f57740a96f0 con 0x7f57740b1b00 2024-03-25T19:14:35.363 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.361+0000 7f576b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:14:35.363 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.361+0000 7f576b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:35.363 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.361+0000 7f576b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:35.363 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.361+0000 7f576b7fe640 10 monclient(hunting): my global_id is 4218 2024-03-25T19:14:35.363 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.361+0000 7f576b7fe640 1 -- 172.21.15.6:0/4186497808 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57740ab0e0 con 0x7f57740b1b00 2024-03-25T19:14:35.366 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.365+0000 7f576b7fe640 1 -- 172.21.15.6:0/4186497808 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3349455994 0 0) 0x7f5774145f70 con 0x7f57740b44b0 2024-03-25T19:14:35.366 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.365+0000 7f576b7fe640 1 -- 172.21.15.6:0/4186497808 >> v1:172.21.15.6:6790/0 conn(0x7f57740b4e70 legacy=0x7f577413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:35.366 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.365+0000 7f576b7fe640 1 -- 172.21.15.6:0/4186497808 >> v1:172.21.15.6:6789/0 conn(0x7f57740b1b00 legacy=0x7f577414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:35.366 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.365+0000 7f576b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:14:35.366 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.365+0000 7f576b7fe640 1 monclient: found mon.noname-c 2024-03-25T19:14:35.366 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.365+0000 7f576b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:14:35.366 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.365+0000 7f576b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:14:35.366 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.365+0000 7f576b7fe640 1 -- 172.21.15.6:0/4186497808 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57740a9b80 con 0x7f57740b44b0 2024-03-25T19:14:35.366 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.365+0000 7f576b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:14:35.367 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.365+0000 7f576b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:14:35.367 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.365+0000 7f576b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:14:35.367 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.365+0000 7f577b1e9640 5 monclient: authenticate success, global_id 4216 2024-03-25T19:14:35.367 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.365+0000 7f577b1e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:14:35.369 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.367+0000 7f576b7fe640 1 -- 172.21.15.6:0/4186497808 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f5764001630 con 0x7f57740b44b0 2024-03-25T19:14:35.369 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.367+0000 7f576b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:35.369 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.367+0000 7f576b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:14:35.369 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.367+0000 7f576b7fe640 10 monclient: dump: 2024-03-25T19:14:35.370 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:14:35.370 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:35.370 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:35.370 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:35.370 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:35.370 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:35.370 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:35.370 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:35.370 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:35.370 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:35.370 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.367+0000 7f577b1e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:14:35.370 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.367+0000 7f576b7fe640 1 -- 172.21.15.6:0/4186497808 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5764001880 con 0x7f57740b44b0 2024-03-25T19:14:35.370 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.367+0000 7f576b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:14:35.371 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.367+0000 7f577b1e9640 10 monclient: get_monmap_and_config success 2024-03-25T19:14:35.371 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.367+0000 7f576b7fe640 1 -- 172.21.15.6:0/4186497808 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f5764001c10 con 0x7f57740b44b0 2024-03-25T19:14:35.371 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.367+0000 7f576b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:35.371 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.367+0000 7f577b1e9640 4 set_mon_vals no callback set 2024-03-25T19:14:35.371 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.367+0000 7f576b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:14:35.371 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.367+0000 7f576b7fe640 10 monclient: dump: 2024-03-25T19:14:35.371 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:14:35.371 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:35.371 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:35.371 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:35.371 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:35.371 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:35.371 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:35.371 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:35.372 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:35.372 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:35.372 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.368+0000 7f577b1e9640 10 monclient: shutdown 2024-03-25T19:14:35.372 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.368+0000 7f577b1e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:14:35.372 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.368+0000 7f577b1e9640 1 -- 172.21.15.6:0/4186497808 >> v1:172.21.15.192:6789/0 conn(0x7f57740b44b0 legacy=0x7f5774145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:35.372 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.368+0000 7f577b1e9640 1 -- 172.21.15.6:0/4186497808 shutdown_connections 2024-03-25T19:14:35.372 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.368+0000 7f577b1e9640 1 -- 172.21.15.6:0/4186497808 >> 172.21.15.6:0/4186497808 conn(0x7f57740617a0 msgr2=0x7f5774061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:14:35.372 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.369+0000 7f577b1e9640 1 -- 172.21.15.6:0/4186497808 shutdown_connections 2024-03-25T19:14:35.372 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.369+0000 7f577b1e9640 1 -- 172.21.15.6:0/4186497808 wait complete. 2024-03-25T19:14:35.372 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.369+0000 7f577b1e9640 10 monclient: build_initial_monmap 2024-03-25T19:14:35.372 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.369+0000 7f577b1e9640 10 monclient: monmap: 2024-03-25T19:14:35.372 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:14:35.372 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:14:35.372 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:14:35.372 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:14:35.373 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:14:35.373 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:14:35.373 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:14:35.373 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:14:35.373 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:14:35.373 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:35.377 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.370+0000 7f577b1e9640 1 librados: starting msgr at 2024-03-25T19:14:35.377 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.370+0000 7f577b1e9640 1 librados: starting objecter 2024-03-25T19:14:35.377 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.370+0000 7f577b1e9640 1 Processor -- start 2024-03-25T19:14:35.377 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.370+0000 7f577b1e9640 1 -- start start 2024-03-25T19:14:35.377 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.370+0000 7f577b1e9640 1 librados: setting wanted keys 2024-03-25T19:14:35.377 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.370+0000 7f577b1e9640 1 librados: calling monclient init 2024-03-25T19:14:35.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.370+0000 7f577b1e9640 10 monclient: init 2024-03-25T19:14:35.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.371+0000 7f577b1e9640 10 monclient: _reopen_session rank -1 2024-03-25T19:14:35.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.371+0000 7f577b1e9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:14:35.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.371+0000 7f577b1e9640 10 monclient(hunting): picked mon.noname-b con 0x7f57740b1b00 addr v1:172.21.15.192:6789/0 2024-03-25T19:14:35.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.371+0000 7f577b1e9640 10 monclient(hunting): picked mon.noname-c con 0x7f57740b44b0 addr v1:172.21.15.6:6790/0 2024-03-25T19:14:35.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.371+0000 7f577b1e9640 10 monclient(hunting): picked mon.noname-a con 0x7f57740b4e70 addr v1:172.21.15.6:6789/0 2024-03-25T19:14:35.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.371+0000 7f577b1e9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57741379d0 con 0x7f57740b4e70 2024-03-25T19:14:35.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.371+0000 7f577b1e9640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5774137cb0 con 0x7f57740b1b00 2024-03-25T19:14:35.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.371+0000 7f577b1e9640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5774136510 con 0x7f57740b44b0 2024-03-25T19:14:35.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.371+0000 7f577b1e9640 10 monclient(hunting): _renew_subs 2024-03-25T19:14:35.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.371+0000 7f577b1e9640 10 monclient(hunting): authenticate will time out at 5979.611719s 2024-03-25T19:14:35.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.372+0000 7f5778f85640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f57740b1b00 0x7f57741358c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:36286/0 (socket says 172.21.15.6:36286) 2024-03-25T19:14:35.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.372+0000 7f5778f85640 1 -- 172.21.15.6:0/33020765 learned_addr learned my addr 172.21.15.6:0/33020765 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:14:35.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.372+0000 7f57697fa640 10 client.?.objecter ms_handle_connect 0x7f57740b44b0 2024-03-25T19:14:35.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.372+0000 7f57697fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:35.379 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.372+0000 7f57697fa640 10 client.?.objecter ms_handle_connect 0x7f57740b1b00 2024-03-25T19:14:35.379 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.372+0000 7f57697fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:35.379 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.372+0000 7f57697fa640 10 client.?.objecter ms_handle_connect 0x7f57740b4e70 2024-03-25T19:14:35.379 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.372+0000 7f57697fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:35.379 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.375+0000 7f57697fa640 1 -- 172.21.15.6:0/33020765 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2619985279 0 0) 0x7f5774137cb0 con 0x7f57740b1b00 2024-03-25T19:14:35.379 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.375+0000 7f57697fa640 10 monclient(hunting): _negotiate 2024-03-25T19:14:35.379 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.375+0000 7f57697fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:35.379 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.375+0000 7f57697fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:35.379 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.375+0000 7f57697fa640 10 monclient(hunting): my global_id is 4219 2024-03-25T19:14:35.379 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.375+0000 7f57697fa640 1 -- 172.21.15.6:0/33020765 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5760004eb0 con 0x7f57740b1b00 2024-03-25T19:14:35.379 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.375+0000 7f57697fa640 1 -- 172.21.15.6:0/33020765 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 494048508 0 0) 0x7f57741379d0 con 0x7f57740b4e70 2024-03-25T19:14:35.379 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.375+0000 7f57697fa640 10 monclient(hunting): _negotiate 2024-03-25T19:14:35.379 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.375+0000 7f57697fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:35.379 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.375+0000 7f57697fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:35.379 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.375+0000 7f57697fa640 10 monclient(hunting): my global_id is 4221 2024-03-25T19:14:35.380 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.375+0000 7f57697fa640 1 -- 172.21.15.6:0/33020765 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5774137cb0 con 0x7f57740b4e70 2024-03-25T19:14:35.380 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.376+0000 7f57697fa640 1 -- 172.21.15.6:0/33020765 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 125528962 0 0) 0x7f5774136510 con 0x7f57740b44b0 2024-03-25T19:14:35.380 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.376+0000 7f57697fa640 10 monclient(hunting): _negotiate 2024-03-25T19:14:35.380 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.376+0000 7f57697fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:35.380 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.376+0000 7f57697fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:35.380 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.376+0000 7f57697fa640 10 monclient(hunting): my global_id is 4220 2024-03-25T19:14:35.380 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.376+0000 7f57697fa640 1 -- 172.21.15.6:0/33020765 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57741379d0 con 0x7f57740b44b0 2024-03-25T19:14:35.383 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.383+0000 7f57697fa640 1 -- 172.21.15.6:0/33020765 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1333434050 0 0) 0x7f5760004eb0 con 0x7f57740b1b00 2024-03-25T19:14:35.383 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.383+0000 7f57697fa640 1 -- 172.21.15.6:0/33020765 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5774136510 con 0x7f57740b1b00 2024-03-25T19:14:35.383 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.383+0000 7f57697fa640 1 -- 172.21.15.6:0/33020765 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f57700018b0 con 0x7f57740b1b00 2024-03-25T19:14:35.383 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.383+0000 7f57697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:35.383 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.383+0000 7f57697fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:14:35.383 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.383+0000 7f57697fa640 10 monclient(hunting): dump: 2024-03-25T19:14:35.383 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:14:35.384 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:35.384 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:35.384 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:35.384 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:35.384 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:35.384 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:35.384 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:35.384 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:35.384 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:35.384 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.383+0000 7f57697fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:14:35.385 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.384+0000 7f57697fa640 1 -- 172.21.15.6:0/33020765 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3087116734 0 0) 0x7f57741379d0 con 0x7f57740b44b0 2024-03-25T19:14:35.386 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.384+0000 7f57697fa640 1 -- 172.21.15.6:0/33020765 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5760004eb0 con 0x7f57740b44b0 2024-03-25T19:14:35.386 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.384+0000 7f57697fa640 1 -- 172.21.15.6:0/33020765 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f5764001b90 con 0x7f57740b44b0 2024-03-25T19:14:35.386 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.384+0000 7f57697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:35.386 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.384+0000 7f57697fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:14:35.386 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.384+0000 7f57697fa640 10 monclient(hunting): dump: 2024-03-25T19:14:35.386 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:14:35.386 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:35.386 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:35.386 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:35.386 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:35.386 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:35.386 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:35.387 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:35.387 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:35.387 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:35.387 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.385+0000 7f57697fa640 1 -- 172.21.15.6:0/33020765 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1838596990 0 0) 0x7f5774137cb0 con 0x7f57740b4e70 2024-03-25T19:14:35.387 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.385+0000 7f57697fa640 1 -- 172.21.15.6:0/33020765 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57741379d0 con 0x7f57740b4e70 2024-03-25T19:14:35.387 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.385+0000 7f57697fa640 1 -- 172.21.15.6:0/33020765 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f576c001b60 con 0x7f57740b4e70 2024-03-25T19:14:35.387 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.385+0000 7f57697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:35.387 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.385+0000 7f57697fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:14:35.387 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.385+0000 7f57697fa640 10 monclient(hunting): dump: 2024-03-25T19:14:35.387 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:14:35.387 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:35.387 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:35.387 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:35.387 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:35.387 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:35.388 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:35.388 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:35.388 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:35.388 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:35.390 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.388+0000 7f57697fa640 1 -- 172.21.15.6:0/33020765 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1915276849 0 0) 0x7f5774136510 con 0x7f57740b1b00 2024-03-25T19:14:35.390 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.388+0000 7f57697fa640 1 -- 172.21.15.6:0/33020765 >> v1:172.21.15.6:6790/0 conn(0x7f57740b44b0 legacy=0x7f5774133e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:35.390 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.388+0000 7f57697fa640 1 -- 172.21.15.6:0/33020765 >> v1:172.21.15.6:6789/0 conn(0x7f57740b4e70 legacy=0x7f57741380d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:35.390 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.388+0000 7f57697fa640 10 monclient: _finish_hunting 0 2024-03-25T19:14:35.390 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.388+0000 7f57697fa640 1 monclient: found mon.b 2024-03-25T19:14:35.390 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.388+0000 7f57697fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:14:35.390 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.388+0000 7f57697fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:35.391 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.388+0000 7f57697fa640 1 -- 172.21.15.6:0/33020765 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5774136a30 con 0x7f57740b1b00 2024-03-25T19:14:35.391 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.388+0000 7f57697fa640 10 monclient: _finish_auth 0 2024-03-25T19:14:35.391 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.388+0000 7f57697fa640 10 monclient: _check_auth_tickets 2024-03-25T19:14:35.391 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.388+0000 7f57697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:14:35.391 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.388+0000 7f577b1e9640 5 monclient: authenticate success, global_id 4219 2024-03-25T19:14:35.391 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.388+0000 7f577b1e9640 10 monclient: _renew_subs 2024-03-25T19:14:35.391 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.388+0000 7f577b1e9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:35.391 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.388+0000 7f577b1e9640 1 -- 172.21.15.6:0/33020765 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5774136080 con 0x7f57740b1b00 2024-03-25T19:14:35.391 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.388+0000 7f577b1e9640 10 client.4219.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:14:35.391 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.388+0000 7f577b1e9640 10 monclient: _renew_subs 2024-03-25T19:14:35.391 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.388+0000 7f577b1e9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:35.391 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.388+0000 7f577b1e9640 1 -- 172.21.15.6:0/33020765 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5774134770 con 0x7f57740b1b00 2024-03-25T19:14:35.391 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.388+0000 7f577b1e9640 1 librados: init done 2024-03-25T19:14:35.391 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.389+0000 7f577b1e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:35.392 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.389+0000 7f577b1e9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:35.392 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.389+0000 7f577b1e9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:35.392 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.389+0000 7f577b1e9640 1 -- 172.21.15.6:0/33020765 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57741366a0 con 0x7f57740b1b00 2024-03-25T19:14:35.418 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.418+0000 7f57697fa640 1 -- 172.21.15.6:0/33020765 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5770003610 con 0x7f57740b1b00 2024-03-25T19:14:35.418 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.418+0000 7f57697fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:14:35.419 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.418+0000 7f57697fa640 1 -- 172.21.15.6:0/33020765 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f57700039a0 con 0x7f57740b1b00 2024-03-25T19:14:35.419 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.418+0000 7f57697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:35.419 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.418+0000 7f57697fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:14:35.419 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.418+0000 7f57697fa640 10 monclient: dump: 2024-03-25T19:14:35.419 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:14:35.419 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:35.419 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:35.419 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:35.419 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:35.419 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:35.419 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:35.419 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:35.419 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:35.420 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:35.420 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.418+0000 7f576affd640 4 set_mon_vals no callback set 2024-03-25T19:14:35.420 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.419+0000 7f57697fa640 1 -- 172.21.15.6:0/33020765 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f5770003bc0 con 0x7f57740b1b00 2024-03-25T19:14:35.420 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.419+0000 7f576bfff640 10 monclient: get_auth_request con 0x7f576006e110 auth_method 0 2024-03-25T19:14:35.421 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.420+0000 7f57697fa640 10 client.4219.objecter ms_handle_connect 0x7f576006e110 2024-03-25T19:14:35.423 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.423+0000 7f57697fa640 1 -- 172.21.15.6:0/33020765 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f57700546d0 con 0x7f57740b1b00 2024-03-25T19:14:35.423 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.423+0000 7f57697fa640 10 client.4219.objecter ms_dispatch 0x7f5774000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:14:35.423 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.423+0000 7f57697fa640 3 client.4219.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:14:35.423 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.423+0000 7f57697fa640 3 client.4219.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:14:35.423 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.423+0000 7f57697fa640 20 client.4219.objecter dump_active .. 0 homeless 2024-03-25T19:14:35.702 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.701+0000 7f57697fa640 1 -- 172.21.15.6:0/33020765 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f57700501e0 con 0x7f57740b1b00 2024-03-25T19:14:35.702 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.702+0000 7f57697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:35.702 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:35.702+0000 7f57697fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:14:36.048 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:36.047+0000 7f577b1e9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:14:36.048 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:36.048+0000 7f577b1e9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:14:36.048 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:36.048+0000 7f577b1e9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:36.048 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:36.048+0000 7f577b1e9640 1 -- 172.21.15.6:0/33020765 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f577404f730 con 0x7f57740b1b00 2024-03-25T19:14:36.063 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:36.063+0000 7f57697fa640 1 -- 172.21.15.6:0/33020765 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f5770052110 con 0x7f57740b1b00 2024-03-25T19:14:36.063 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:36.063+0000 7f57697fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:14:36.064 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:36.063+0000 7f57697fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:14:36.064 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:14:36.064 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:14:36.067 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:36.066+0000 7f577b1e9640 10 librados: watch_flush enter 2024-03-25T19:14:36.067 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:36.066+0000 7f577b1e9640 10 librados: watch_flush exit 2024-03-25T19:14:36.067 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:36.066+0000 7f577b1e9640 20 client.4219.objecter shutdown clearing up homeless session... 2024-03-25T19:14:36.067 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:36.066+0000 7f577b1e9640 10 client.4219.objecter successfully canceled tick 2024-03-25T19:14:36.067 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:36.066+0000 7f577b1e9640 1 -- 172.21.15.6:0/33020765 >> v1:172.21.15.6:6800/2936652650 conn(0x7f576006e110 legacy=0x7f57600705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:36.067 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:36.066+0000 7f577b1e9640 10 monclient: shutdown 2024-03-25T19:14:36.067 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:36.066+0000 7f577b1e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:14:36.068 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:36.066+0000 7f577b1e9640 1 -- 172.21.15.6:0/33020765 >> v1:172.21.15.192:6789/0 conn(0x7f57740b1b00 legacy=0x7f57741358c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:36.068 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:36.066+0000 7f577b1e9640 1 -- 172.21.15.6:0/33020765 shutdown_connections 2024-03-25T19:14:36.068 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:36.066+0000 7f577b1e9640 1 -- 172.21.15.6:0/33020765 >> 172.21.15.6:0/33020765 conn(0x7f57740617a0 msgr2=0x7f57740b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:14:36.068 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:36.066+0000 7f577b1e9640 1 -- 172.21.15.6:0/33020765 shutdown_connections 2024-03-25T19:14:36.068 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:36.066+0000 7f577b1e9640 1 -- 172.21.15.6:0/33020765 wait complete. 2024-03-25T19:14:36.068 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:36.066+0000 7f577b1e9640 1 librados: shutdown 2024-03-25T19:14:36.080 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:14:36.773 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:26.254 35209== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:37.021 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:26.502 35208== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:37.180 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:26.660 35210== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:37.180 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:26.646 35211== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:37.688 INFO:tasks.ceph.osd.0.smithi006.stderr:==00:00:00:27.181 35451== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:38.108 INFO:tasks.ceph.osd.2.smithi006.stderr:==00:00:00:27.592 35452== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:38.165 INFO:tasks.ceph.osd.3.smithi006.stderr:==00:00:00:27.634 35453== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:38.524 INFO:tasks.ceph.osd.1.smithi006.stderr:==00:00:00:27.993 35454== Warning: unimplemented fcntl command: 1038 2024-03-25T19:14:42.886 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:14:43.027 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.027+0000 7fc7e264a640 10 monclient: get_monmap_and_config 2024-03-25T19:14:43.028 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.027+0000 7fc7e264a640 10 monclient: build_initial_monmap 2024-03-25T19:14:43.028 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.027+0000 7fc7e264a640 10 monclient: monmap: 2024-03-25T19:14:43.028 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:14:43.028 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:43.028 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:14:43.028631+0000 2024-03-25T19:14:43.028 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:14:43.028631+0000 2024-03-25T19:14:43.028 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:14:43.028 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:43.028 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:14:43.028 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:14:43.028 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:14:43.028 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:43.030 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.028+0000 7fc7e264a640 1 Processor -- start 2024-03-25T19:14:43.030 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.028+0000 7fc7e264a640 1 -- start start 2024-03-25T19:14:43.030 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.028+0000 7fc7e264a640 10 monclient: init 2024-03-25T19:14:43.030 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.029+0000 7fc7e264a640 10 monclient: _reopen_session rank -1 2024-03-25T19:14:43.030 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.029+0000 7fc7e264a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:14:43.031 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.029+0000 7fc7e264a640 10 monclient(hunting): picked mon.noname-c con 0x7fc7dc0b4e70 addr v1:172.21.15.192:6789/0 2024-03-25T19:14:43.031 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.029+0000 7fc7e264a640 10 monclient(hunting): picked mon.noname-b con 0x7fc7dc0b44b0 addr v1:172.21.15.6:6790/0 2024-03-25T19:14:43.031 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.029+0000 7fc7e264a640 10 monclient(hunting): picked mon.noname-a con 0x7fc7dc0b1b00 addr v1:172.21.15.6:6789/0 2024-03-25T19:14:43.031 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.029+0000 7fc7e264a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7dc0a96f0 con 0x7fc7dc0b1b00 2024-03-25T19:14:43.031 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.030+0000 7fc7e264a640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7dc0ab0e0 con 0x7fc7dc0b4e70 2024-03-25T19:14:43.031 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.030+0000 7fc7e264a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7dc145f70 con 0x7fc7dc0b44b0 2024-03-25T19:14:43.031 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.030+0000 7fc7e264a640 10 monclient(hunting): _renew_subs 2024-03-25T19:14:43.031 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.030+0000 7fc7e264a640 10 monclient(hunting): authenticate will time out at 5987.270018s 2024-03-25T19:14:43.031 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.030+0000 7fc7e0be7640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc7dc0b1b00 0x7fc7dc14dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41308/0 (socket says 172.21.15.6:41308) 2024-03-25T19:14:43.031 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.030+0000 7fc7e0be7640 1 -- 172.21.15.6:0/2741302753 learned_addr learned my addr 172.21.15.6:0/2741302753 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:14:43.035 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.033+0000 7fc7daffd640 1 -- 172.21.15.6:0/2741302753 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 538446387 0 0) 0x7fc7dc145f70 con 0x7fc7dc0b44b0 2024-03-25T19:14:43.035 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.034+0000 7fc7daffd640 10 monclient(hunting): _negotiate 2024-03-25T19:14:43.035 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.034+0000 7fc7daffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:43.035 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.034+0000 7fc7daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:43.035 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.034+0000 7fc7daffd640 10 monclient(hunting): my global_id is 4223 2024-03-25T19:14:43.035 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.034+0000 7fc7daffd640 1 -- 172.21.15.6:0/2741302753 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7c4003520 con 0x7fc7dc0b44b0 2024-03-25T19:14:43.035 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.034+0000 7fc7daffd640 1 -- 172.21.15.6:0/2741302753 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3273085273 0 0) 0x7fc7dc0ab0e0 con 0x7fc7dc0b4e70 2024-03-25T19:14:43.036 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.034+0000 7fc7daffd640 10 monclient(hunting): _negotiate 2024-03-25T19:14:43.036 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.034+0000 7fc7daffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:43.036 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.034+0000 7fc7daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:43.036 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.034+0000 7fc7daffd640 10 monclient(hunting): my global_id is 4222 2024-03-25T19:14:43.036 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.034+0000 7fc7daffd640 1 -- 172.21.15.6:0/2741302753 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7dc145f70 con 0x7fc7dc0b4e70 2024-03-25T19:14:43.036 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.034+0000 7fc7daffd640 1 -- 172.21.15.6:0/2741302753 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1912938459 0 0) 0x7fc7dc0a96f0 con 0x7fc7dc0b1b00 2024-03-25T19:14:43.036 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.034+0000 7fc7daffd640 10 monclient(hunting): _negotiate 2024-03-25T19:14:43.036 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.034+0000 7fc7daffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:43.036 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.034+0000 7fc7daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:43.036 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.035+0000 7fc7daffd640 10 monclient(hunting): my global_id is 4224 2024-03-25T19:14:43.036 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.035+0000 7fc7daffd640 1 -- 172.21.15.6:0/2741302753 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7dc0ab0e0 con 0x7fc7dc0b1b00 2024-03-25T19:14:43.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.039+0000 7fc7daffd640 1 -- 172.21.15.6:0/2741302753 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3781955666 0 0) 0x7fc7dc145f70 con 0x7fc7dc0b4e70 2024-03-25T19:14:43.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.039+0000 7fc7daffd640 1 -- 172.21.15.6:0/2741302753 >> v1:172.21.15.6:6790/0 conn(0x7fc7dc0b44b0 legacy=0x7fc7dc145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:43.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.039+0000 7fc7daffd640 1 -- 172.21.15.6:0/2741302753 >> v1:172.21.15.6:6789/0 conn(0x7fc7dc0b1b00 legacy=0x7fc7dc14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:43.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.039+0000 7fc7daffd640 10 monclient: _finish_hunting 0 2024-03-25T19:14:43.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.039+0000 7fc7daffd640 1 monclient: found mon.noname-c 2024-03-25T19:14:43.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.039+0000 7fc7daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:14:43.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.039+0000 7fc7daffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:14:43.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.039+0000 7fc7daffd640 1 -- 172.21.15.6:0/2741302753 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7dc0a9b80 con 0x7fc7dc0b4e70 2024-03-25T19:14:43.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.039+0000 7fc7daffd640 10 monclient: _finish_auth 0 2024-03-25T19:14:43.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.039+0000 7fc7daffd640 10 monclient: _check_auth_tickets 2024-03-25T19:14:43.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.039+0000 7fc7daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:14:43.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.039+0000 7fc7daffd640 1 -- 172.21.15.6:0/2741302753 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fc7cc001670 con 0x7fc7dc0b4e70 2024-03-25T19:14:43.042 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.039+0000 7fc7daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:43.042 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.039+0000 7fc7daffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:14:43.042 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.039+0000 7fc7daffd640 10 monclient: dump: 2024-03-25T19:14:43.042 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:14:43.042 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:43.042 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:43.042 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:43.042 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:43.042 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:43.042 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:43.042 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:43.042 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:43.042 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:43.043 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.040+0000 7fc7e264a640 5 monclient: authenticate success, global_id 4222 2024-03-25T19:14:43.043 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.040+0000 7fc7e264a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:14:43.046 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.041+0000 7fc7daffd640 1 -- 172.21.15.6:0/2741302753 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7cc0017d0 con 0x7fc7dc0b4e70 2024-03-25T19:14:43.046 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.041+0000 7fc7daffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:14:43.046 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.041+0000 7fc7e264a640 10 monclient: get_monmap_and_config success 2024-03-25T19:14:43.046 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.041+0000 7fc7e264a640 4 set_mon_vals no callback set 2024-03-25T19:14:43.047 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.042+0000 7fc7e264a640 10 monclient: shutdown 2024-03-25T19:14:43.047 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.042+0000 7fc7e264a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:14:43.047 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.042+0000 7fc7e264a640 1 -- 172.21.15.6:0/2741302753 >> v1:172.21.15.192:6789/0 conn(0x7fc7dc0b4e70 legacy=0x7fc7dc13f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:43.047 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.042+0000 7fc7e264a640 1 -- 172.21.15.6:0/2741302753 shutdown_connections 2024-03-25T19:14:43.047 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.042+0000 7fc7e264a640 1 -- 172.21.15.6:0/2741302753 >> 172.21.15.6:0/2741302753 conn(0x7fc7dc0617a0 msgr2=0x7fc7dc061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:14:43.047 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.042+0000 7fc7e264a640 1 -- 172.21.15.6:0/2741302753 shutdown_connections 2024-03-25T19:14:43.047 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.042+0000 7fc7e264a640 1 -- 172.21.15.6:0/2741302753 wait complete. 2024-03-25T19:14:43.047 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.043+0000 7fc7e264a640 10 monclient: build_initial_monmap 2024-03-25T19:14:43.047 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.043+0000 7fc7e264a640 10 monclient: monmap: 2024-03-25T19:14:43.047 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:14:43.047 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:14:43.047 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:14:43.047 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:14:43.047 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:14:43.048 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:14:43.048 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:14:43.048 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:14:43.048 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:14:43.048 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:43.048 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.044+0000 7fc7e264a640 1 librados: starting msgr at 2024-03-25T19:14:43.048 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.044+0000 7fc7e264a640 1 librados: starting objecter 2024-03-25T19:14:43.048 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.044+0000 7fc7e264a640 1 Processor -- start 2024-03-25T19:14:43.048 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.044+0000 7fc7e264a640 1 -- start start 2024-03-25T19:14:43.048 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.044+0000 7fc7e264a640 1 librados: setting wanted keys 2024-03-25T19:14:43.048 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.044+0000 7fc7e264a640 1 librados: calling monclient init 2024-03-25T19:14:43.048 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.044+0000 7fc7e264a640 10 monclient: init 2024-03-25T19:14:43.048 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.045+0000 7fc7e264a640 10 monclient: _reopen_session rank -1 2024-03-25T19:14:43.048 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.045+0000 7fc7e264a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:14:43.048 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.045+0000 7fc7e264a640 10 monclient(hunting): picked mon.noname-a con 0x7fc7dc0b1b00 addr v1:172.21.15.6:6789/0 2024-03-25T19:14:43.048 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.045+0000 7fc7e264a640 10 monclient(hunting): picked mon.noname-c con 0x7fc7dc0b44b0 addr v1:172.21.15.6:6790/0 2024-03-25T19:14:43.049 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.045+0000 7fc7e264a640 10 monclient(hunting): picked mon.noname-b con 0x7fc7dc0b4e70 addr v1:172.21.15.192:6789/0 2024-03-25T19:14:43.049 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.046+0000 7fc7e264a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7dc135eb0 con 0x7fc7dc0b1b00 2024-03-25T19:14:43.049 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.046+0000 7fc7e264a640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7dc136b50 con 0x7fc7dc0b4e70 2024-03-25T19:14:43.049 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.046+0000 7fc7e264a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7dc1378b0 con 0x7fc7dc0b44b0 2024-03-25T19:14:43.049 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.046+0000 7fc7e264a640 10 monclient(hunting): _renew_subs 2024-03-25T19:14:43.049 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.046+0000 7fc7e264a640 10 monclient(hunting): authenticate will time out at 5987.286207s 2024-03-25T19:14:43.049 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.046+0000 7fc7db7fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fc7dc0b44b0 0x7fc7dc1357a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:57962/0 (socket says 172.21.15.6:57962) 2024-03-25T19:14:43.049 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.046+0000 7fc7db7fe640 1 -- 172.21.15.6:0/3394348646 learned_addr learned my addr 172.21.15.6:0/3394348646 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:14:43.049 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.046+0000 7fc7e0be7640 1 --1- 172.21.15.6:0/3394348646 >> v1:172.21.15.192:6789/0 conn(0x7fc7dc0b4e70 0x7fc7dc136440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:47454/0 (socket says 172.21.15.6:47454) 2024-03-25T19:14:43.049 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.046+0000 7fc7d8ff9640 10 client.?.objecter ms_handle_connect 0x7fc7dc0b44b0 2024-03-25T19:14:43.049 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.046+0000 7fc7d8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:43.049 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.047+0000 7fc7d8ff9640 10 client.?.objecter ms_handle_connect 0x7fc7dc0b4e70 2024-03-25T19:14:43.049 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.047+0000 7fc7d8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:43.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.050+0000 7fc7d8ff9640 1 -- 172.21.15.6:0/3394348646 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4144381118 0 0) 0x7fc7dc1378b0 con 0x7fc7dc0b44b0 2024-03-25T19:14:43.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.050+0000 7fc7d8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:14:43.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.050+0000 7fc7d8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:43.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.050+0000 7fc7d8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:43.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.050+0000 7fc7d8ff9640 10 monclient(hunting): my global_id is 4226 2024-03-25T19:14:43.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.050+0000 7fc7d8ff9640 1 -- 172.21.15.6:0/3394348646 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7c8004eb0 con 0x7fc7dc0b44b0 2024-03-25T19:14:43.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.050+0000 7fc7d8ff9640 1 -- 172.21.15.6:0/3394348646 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2662533439 0 0) 0x7fc7dc136b50 con 0x7fc7dc0b4e70 2024-03-25T19:14:43.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.050+0000 7fc7d8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:14:43.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.050+0000 7fc7d8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:43.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.050+0000 7fc7d8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:43.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.051+0000 7fc7d8ff9640 10 monclient(hunting): my global_id is 4225 2024-03-25T19:14:43.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.051+0000 7fc7d8ff9640 1 -- 172.21.15.6:0/3394348646 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7dc1378b0 con 0x7fc7dc0b4e70 2024-03-25T19:14:43.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.058+0000 7fc7d8ff9640 1 -- 172.21.15.6:0/3394348646 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3895157262 0 0) 0x7fc7dc1378b0 con 0x7fc7dc0b4e70 2024-03-25T19:14:43.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.058+0000 7fc7d8ff9640 1 -- 172.21.15.6:0/3394348646 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7dc136b50 con 0x7fc7dc0b4e70 2024-03-25T19:14:43.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.058+0000 7fc7d8ff9640 1 -- 172.21.15.6:0/3394348646 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fc7d4001c70 con 0x7fc7dc0b4e70 2024-03-25T19:14:43.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.058+0000 7fc7d8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:43.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.058+0000 7fc7d8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:14:43.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.058+0000 7fc7d8ff9640 10 monclient(hunting): dump: 2024-03-25T19:14:43.060 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:14:43.060 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:43.060 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:43.060 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:43.060 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:43.060 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:43.060 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:43.060 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:43.060 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:43.060 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:43.060 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.058+0000 7fc7d8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:14:43.060 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.059+0000 7fc7d8ff9640 1 -- 172.21.15.6:0/3394348646 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 900061668 0 0) 0x7fc7c8004eb0 con 0x7fc7dc0b44b0 2024-03-25T19:14:43.061 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.059+0000 7fc7d8ff9640 1 -- 172.21.15.6:0/3394348646 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7dc1378b0 con 0x7fc7dc0b44b0 2024-03-25T19:14:43.061 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.059+0000 7fc7d8ff9640 1 -- 172.21.15.6:0/3394348646 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fc7d00018d0 con 0x7fc7dc0b44b0 2024-03-25T19:14:43.061 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.059+0000 7fc7d8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:43.061 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.059+0000 7fc7d8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:14:43.061 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.059+0000 7fc7d8ff9640 10 monclient(hunting): dump: 2024-03-25T19:14:43.061 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:14:43.061 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:43.061 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:43.061 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:43.061 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:43.061 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:43.061 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:43.062 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:43.062 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:43.062 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:43.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.064+0000 7fc7d8ff9640 1 -- 172.21.15.6:0/3394348646 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 391696654 0 0) 0x7fc7dc136b50 con 0x7fc7dc0b4e70 2024-03-25T19:14:43.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.064+0000 7fc7d8ff9640 1 -- 172.21.15.6:0/3394348646 >> v1:172.21.15.6:6790/0 conn(0x7fc7dc0b44b0 legacy=0x7fc7dc1357a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:43.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.064+0000 7fc7d8ff9640 1 -- 172.21.15.6:0/3394348646 >> v1:172.21.15.6:6789/0 conn(0x7fc7dc0b1b00 legacy=0x7fc7dc137fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:43.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.064+0000 7fc7d8ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:14:43.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.064+0000 7fc7d8ff9640 1 monclient: found mon.b 2024-03-25T19:14:43.066 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.064+0000 7fc7d8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:14:43.066 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.064+0000 7fc7d8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:43.066 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.064+0000 7fc7d8ff9640 1 -- 172.21.15.6:0/3394348646 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7dc137b90 con 0x7fc7dc0b4e70 2024-03-25T19:14:43.066 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.064+0000 7fc7d8ff9640 10 monclient: _finish_auth 0 2024-03-25T19:14:43.066 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.064+0000 7fc7d8ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:14:43.066 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.064+0000 7fc7d8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:14:43.066 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.064+0000 7fc7e264a640 5 monclient: authenticate success, global_id 4225 2024-03-25T19:14:43.066 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.064+0000 7fc7e264a640 10 monclient: _renew_subs 2024-03-25T19:14:43.066 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.064+0000 7fc7e264a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:43.066 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.065+0000 7fc7e264a640 1 -- 172.21.15.6:0/3394348646 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7dc138ff0 con 0x7fc7dc0b4e70 2024-03-25T19:14:43.066 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.065+0000 7fc7e264a640 10 client.4225.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:14:43.066 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.065+0000 7fc7e264a640 10 monclient: _renew_subs 2024-03-25T19:14:43.066 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.065+0000 7fc7e264a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:43.066 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.065+0000 7fc7e264a640 1 -- 172.21.15.6:0/3394348646 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7dc1e3210 con 0x7fc7dc0b4e70 2024-03-25T19:14:43.067 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.065+0000 7fc7e264a640 1 librados: init done 2024-03-25T19:14:43.067 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.066+0000 7fc7e264a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:43.067 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.066+0000 7fc7e264a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:43.067 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.066+0000 7fc7e264a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:43.067 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.066+0000 7fc7e264a640 1 -- 172.21.15.6:0/3394348646 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7dc04f730 con 0x7fc7dc0b4e70 2024-03-25T19:14:43.091 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.091+0000 7fc7d8ff9640 1 -- 172.21.15.6:0/3394348646 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7d4004b60 con 0x7fc7dc0b4e70 2024-03-25T19:14:43.091 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.091+0000 7fc7d8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:14:43.091 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.091+0000 7fc7d8ff9640 1 -- 172.21.15.6:0/3394348646 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fc7d4004eb0 con 0x7fc7dc0b4e70 2024-03-25T19:14:43.092 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.091+0000 7fc7d8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:43.092 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.091+0000 7fc7d8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:14:43.092 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.091+0000 7fc7d8ff9640 10 monclient: dump: 2024-03-25T19:14:43.092 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:14:43.092 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:43.092 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:43.092 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:43.092 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:43.092 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:43.092 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:43.092 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:43.092 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:43.092 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:43.093 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.091+0000 7fc7da7fc640 4 set_mon_vals no callback set 2024-03-25T19:14:43.093 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.092+0000 7fc7d8ff9640 1 -- 172.21.15.6:0/3394348646 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7fc7d40050d0 con 0x7fc7dc0b4e70 2024-03-25T19:14:43.093 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.093+0000 7fc7dbfff640 10 monclient: get_auth_request con 0x7fc7c806de30 auth_method 0 2024-03-25T19:14:43.093 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.093+0000 7fc7d8ff9640 10 client.4225.objecter ms_handle_connect 0x7fc7c806de30 2024-03-25T19:14:43.346 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.346+0000 7fc7d8ff9640 1 -- 172.21.15.6:0/3394348646 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7fc7d4054a20 con 0x7fc7dc0b4e70 2024-03-25T19:14:43.346 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.346+0000 7fc7d8ff9640 10 client.4225.objecter ms_dispatch 0x7fc7dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:14:43.346 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.346+0000 7fc7d8ff9640 3 client.4225.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:14:43.346 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.346+0000 7fc7d8ff9640 3 client.4225.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:14:43.346 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.346+0000 7fc7d8ff9640 20 client.4225.objecter dump_active .. 0 homeless 2024-03-25T19:14:43.348 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.348+0000 7fc7d8ff9640 1 -- 172.21.15.6:0/3394348646 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc7d4050530 con 0x7fc7dc0b4e70 2024-03-25T19:14:43.348 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.348+0000 7fc7d8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:43.348 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.348+0000 7fc7d8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:14:43.593 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.593+0000 7fc7e264a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:14:43.594 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.593+0000 7fc7e264a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:14:43.594 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.593+0000 7fc7e264a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:43.594 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.593+0000 7fc7e264a640 1 -- 172.21.15.6:0/3394348646 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7dc0aa090 con 0x7fc7dc0b4e70 2024-03-25T19:14:43.618 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.618+0000 7fc7d8ff9640 1 -- 172.21.15.6:0/3394348646 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7fc7d4052460 con 0x7fc7dc0b4e70 2024-03-25T19:14:43.619 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.618+0000 7fc7d8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:14:43.619 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.618+0000 7fc7d8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:14:43.619 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:14:43.619 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:14:43.621 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.620+0000 7fc7e264a640 10 librados: watch_flush enter 2024-03-25T19:14:43.621 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.620+0000 7fc7e264a640 10 librados: watch_flush exit 2024-03-25T19:14:43.621 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.620+0000 7fc7e264a640 20 client.4225.objecter shutdown clearing up homeless session... 2024-03-25T19:14:43.621 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.620+0000 7fc7e264a640 10 client.4225.objecter successfully canceled tick 2024-03-25T19:14:43.621 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.620+0000 7fc7e264a640 1 -- 172.21.15.6:0/3394348646 >> v1:172.21.15.6:6800/2936652650 conn(0x7fc7c806de30 legacy=0x7fc7c80702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:43.621 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.620+0000 7fc7e264a640 10 monclient: shutdown 2024-03-25T19:14:43.621 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.620+0000 7fc7e264a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:14:43.621 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.620+0000 7fc7e264a640 1 -- 172.21.15.6:0/3394348646 >> v1:172.21.15.192:6789/0 conn(0x7fc7dc0b4e70 legacy=0x7fc7dc136440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:43.621 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.620+0000 7fc7e264a640 1 -- 172.21.15.6:0/3394348646 shutdown_connections 2024-03-25T19:14:43.621 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.620+0000 7fc7e264a640 1 -- 172.21.15.6:0/3394348646 >> 172.21.15.6:0/3394348646 conn(0x7fc7dc0617a0 msgr2=0x7fc7dc0b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:14:43.621 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.621+0000 7fc7e264a640 1 -- 172.21.15.6:0/3394348646 shutdown_connections 2024-03-25T19:14:43.621 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.621+0000 7fc7e264a640 1 -- 172.21.15.6:0/3394348646 wait complete. 2024-03-25T19:14:43.621 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:43.621+0000 7fc7e264a640 1 librados: shutdown 2024-03-25T19:14:43.631 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:14:45.062 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-03-25T19:14:45.059+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T19:14:45.469 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-03-25T19:14:45.466+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T19:14:45.720 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-03-25T19:14:45.717+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T19:14:45.968 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-03-25T19:14:45.959+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T19:14:46.702 INFO:tasks.ceph.osd.3.smithi006.stderr:2024-03-25T19:14:46.699+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T19:14:47.048 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-03-25T19:14:47.045+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T19:14:47.070 INFO:tasks.ceph.osd.2.smithi006.stderr:2024-03-25T19:14:47.067+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T19:14:48.306 INFO:tasks.ceph.osd.1.smithi006.stderr:2024-03-25T19:14:48.303+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T19:14:50.439 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:14:50.561 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.561+0000 7f7e85a3e640 10 monclient: get_monmap_and_config 2024-03-25T19:14:50.561 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.561+0000 7f7e85a3e640 10 monclient: build_initial_monmap 2024-03-25T19:14:50.561 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.561+0000 7f7e85a3e640 10 monclient: monmap: 2024-03-25T19:14:50.561 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:14:50.561 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:50.561 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:14:50.562171+0000 2024-03-25T19:14:50.561 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:14:50.562171+0000 2024-03-25T19:14:50.561 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:14:50.561 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:50.561 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:14:50.561 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:14:50.561 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:14:50.562 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:50.563 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.561+0000 7f7e85a3e640 1 Processor -- start 2024-03-25T19:14:50.563 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.561+0000 7f7e85a3e640 1 -- start start 2024-03-25T19:14:50.563 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.561+0000 7f7e85a3e640 10 monclient: init 2024-03-25T19:14:50.563 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.562+0000 7f7e85a3e640 10 monclient: _reopen_session rank -1 2024-03-25T19:14:50.563 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.562+0000 7f7e85a3e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:14:50.563 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.562+0000 7f7e85a3e640 10 monclient(hunting): picked mon.noname-a con 0x7f7e8006f460 addr v1:172.21.15.6:6789/0 2024-03-25T19:14:50.563 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.562+0000 7f7e85a3e640 10 monclient(hunting): picked mon.noname-b con 0x7f7e8006f840 addr v1:172.21.15.6:6790/0 2024-03-25T19:14:50.563 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.562+0000 7f7e85a3e640 10 monclient(hunting): picked mon.noname-c con 0x7f7e800728f0 addr v1:172.21.15.192:6789/0 2024-03-25T19:14:50.563 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.562+0000 7f7e85a3e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e80071a40 con 0x7f7e8006f460 2024-03-25T19:14:50.563 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.562+0000 7f7e85a3e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e80071db0 con 0x7f7e800728f0 2024-03-25T19:14:50.564 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.562+0000 7f7e85a3e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e80070650 con 0x7f7e8006f840 2024-03-25T19:14:50.564 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.562+0000 7f7e85a3e640 10 monclient(hunting): _renew_subs 2024-03-25T19:14:50.564 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.563+0000 7f7e85a3e640 10 monclient(hunting): authenticate will time out at 5994.802781s 2024-03-25T19:14:50.567 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.567+0000 7f7e7e7fc640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f7e8006f840 0x7f7e800721e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:56232/0 (socket says 172.21.15.6:56232) 2024-03-25T19:14:50.567 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.567+0000 7f7e7e7fc640 1 -- 172.21.15.6:0/4144675692 learned_addr learned my addr 172.21.15.6:0/4144675692 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:14:50.574 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.574+0000 7f7e7dffb640 1 -- 172.21.15.6:0/4144675692 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3769530411 0 0) 0x7f7e80070650 con 0x7f7e8006f840 2024-03-25T19:14:50.574 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.574+0000 7f7e7dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:14:50.574 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.574+0000 7f7e7dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:50.574 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.574+0000 7f7e7dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:50.574 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.574+0000 7f7e7dffb640 10 monclient(hunting): my global_id is 4253 2024-03-25T19:14:50.574 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.574+0000 7f7e7dffb640 1 -- 172.21.15.6:0/4144675692 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e68003520 con 0x7f7e8006f840 2024-03-25T19:14:50.587 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.587+0000 7f7e7dffb640 1 -- 172.21.15.6:0/4144675692 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4126311349 0 0) 0x7f7e68003520 con 0x7f7e8006f840 2024-03-25T19:14:50.587 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.587+0000 7f7e7dffb640 1 -- 172.21.15.6:0/4144675692 >> v1:172.21.15.192:6789/0 conn(0x7f7e800728f0 legacy=0x7f7e8014a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:50.588 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.587+0000 7f7e7dffb640 1 -- 172.21.15.6:0/4144675692 >> v1:172.21.15.6:6789/0 conn(0x7f7e8006f460 legacy=0x7f7e80142cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:14:50.588 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.587+0000 7f7e7dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:14:50.588 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.587+0000 7f7e7dffb640 1 monclient: found mon.noname-b 2024-03-25T19:14:50.588 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.587+0000 7f7e7dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:14:50.588 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.587+0000 7f7e7dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-03-25T19:14:50.588 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.587+0000 7f7e7dffb640 1 -- 172.21.15.6:0/4144675692 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e80070200 con 0x7f7e8006f840 2024-03-25T19:14:50.588 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.587+0000 7f7e7dffb640 10 monclient: _finish_auth 0 2024-03-25T19:14:50.588 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.587+0000 7f7e7dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:14:50.588 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.587+0000 7f7e7dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:14:50.588 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.587+0000 7f7e7dffb640 1 -- 172.21.15.6:0/4144675692 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f7e74001690 con 0x7f7e8006f840 2024-03-25T19:14:50.588 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.587+0000 7f7e7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:50.588 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.587+0000 7f7e7dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:14:50.588 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.587+0000 7f7e7dffb640 10 monclient: dump: 2024-03-25T19:14:50.588 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:14:50.589 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:50.589 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:50.589 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:50.589 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:50.589 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:50.589 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:50.589 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:50.589 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:50.589 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:50.589 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.587+0000 7f7e85a3e640 5 monclient: authenticate success, global_id 4253 2024-03-25T19:14:50.589 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.587+0000 7f7e85a3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:14:50.593 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.593+0000 7f7e7dffb640 1 -- 172.21.15.6:0/4144675692 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e740017f0 con 0x7f7e8006f840 2024-03-25T19:14:50.593 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.593+0000 7f7e7dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:14:50.593 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.593+0000 7f7e7dffb640 1 -- 172.21.15.6:0/4144675692 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f7e74001c40 con 0x7f7e8006f840 2024-03-25T19:14:50.593 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.593+0000 7f7e85a3e640 10 monclient: get_monmap_and_config success 2024-03-25T19:14:50.593 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.593+0000 7f7e85a3e640 4 set_mon_vals no callback set 2024-03-25T19:14:50.593 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.593+0000 7f7e7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:50.593 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.593+0000 7f7e7dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:14:50.593 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.593+0000 7f7e7dffb640 10 monclient: dump: 2024-03-25T19:14:50.593 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:14:50.594 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:50.594 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:50.594 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:50.594 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:50.594 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:50.594 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:50.594 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:50.594 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:50.594 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:50.596 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.594+0000 7f7e85a3e640 10 monclient: shutdown 2024-03-25T19:14:50.596 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.594+0000 7f7e85a3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:14:50.596 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.594+0000 7f7e85a3e640 1 -- 172.21.15.6:0/4144675692 >> v1:172.21.15.6:6790/0 conn(0x7f7e8006f840 legacy=0x7f7e800721e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:50.596 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.594+0000 7f7e85a3e640 1 -- 172.21.15.6:0/4144675692 shutdown_connections 2024-03-25T19:14:50.596 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.594+0000 7f7e85a3e640 1 -- 172.21.15.6:0/4144675692 >> 172.21.15.6:0/4144675692 conn(0x7f7e800b4bd0 msgr2=0x7f7e800b4fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:14:50.596 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.594+0000 7f7e85a3e640 1 -- 172.21.15.6:0/4144675692 shutdown_connections 2024-03-25T19:14:50.596 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.594+0000 7f7e85a3e640 1 -- 172.21.15.6:0/4144675692 wait complete. 2024-03-25T19:14:50.596 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.595+0000 7f7e85a3e640 10 monclient: build_initial_monmap 2024-03-25T19:14:50.596 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.595+0000 7f7e85a3e640 10 monclient: monmap: 2024-03-25T19:14:50.596 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:14:50.596 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:14:50.596 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:14:50.597 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:14:50.597 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:14:50.597 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:14:50.597 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:14:50.597 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:14:50.597 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:14:50.597 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:50.597 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.595+0000 7f7e85a3e640 1 librados: starting msgr at 2024-03-25T19:14:50.597 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.595+0000 7f7e85a3e640 1 librados: starting objecter 2024-03-25T19:14:50.597 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.595+0000 7f7e85a3e640 1 Processor -- start 2024-03-25T19:14:50.597 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.595+0000 7f7e85a3e640 1 -- start start 2024-03-25T19:14:50.597 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.595+0000 7f7e85a3e640 1 librados: setting wanted keys 2024-03-25T19:14:50.597 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.595+0000 7f7e85a3e640 1 librados: calling monclient init 2024-03-25T19:14:50.597 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.595+0000 7f7e85a3e640 10 monclient: init 2024-03-25T19:14:50.598 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.596+0000 7f7e85a3e640 10 monclient: _reopen_session rank -1 2024-03-25T19:14:50.598 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.596+0000 7f7e85a3e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:14:50.598 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.596+0000 7f7e85a3e640 10 monclient(hunting): picked mon.noname-c con 0x7f7e8006f460 addr v1:172.21.15.6:6790/0 2024-03-25T19:14:50.598 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.596+0000 7f7e85a3e640 10 monclient(hunting): picked mon.noname-b con 0x7f7e8006f840 addr v1:172.21.15.192:6789/0 2024-03-25T19:14:50.598 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.596+0000 7f7e85a3e640 10 monclient(hunting): picked mon.noname-a con 0x7f7e800728f0 addr v1:172.21.15.6:6789/0 2024-03-25T19:14:50.598 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.596+0000 7f7e85a3e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e801d4f80 con 0x7f7e800728f0 2024-03-25T19:14:50.598 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.596+0000 7f7e85a3e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e8011ebf0 con 0x7f7e8006f840 2024-03-25T19:14:50.598 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.596+0000 7f7e85a3e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e8011ee40 con 0x7f7e8006f460 2024-03-25T19:14:50.598 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.596+0000 7f7e85a3e640 10 monclient(hunting): _renew_subs 2024-03-25T19:14:50.598 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.596+0000 7f7e85a3e640 10 monclient(hunting): authenticate will time out at 5994.835963s 2024-03-25T19:14:50.608 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.608+0000 7f7e7e7fc640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f7e8006f840 0x7f7e8013ae00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:47750/0 (socket says 172.21.15.6:47750) 2024-03-25T19:14:50.608 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.608+0000 7f7e7e7fc640 1 -- 172.21.15.6:0/3345558652 learned_addr learned my addr 172.21.15.6:0/3345558652 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:14:50.609 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.609+0000 7f7e67fff640 10 client.?.objecter ms_handle_connect 0x7f7e8006f840 2024-03-25T19:14:50.609 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.609+0000 7f7e67fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:50.610 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.610+0000 7f7e67fff640 10 client.?.objecter ms_handle_connect 0x7f7e8006f460 2024-03-25T19:14:50.610 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.610+0000 7f7e67fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:50.619 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.619+0000 7f7e67fff640 10 client.?.objecter ms_handle_connect 0x7f7e800728f0 2024-03-25T19:14:50.619 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.619+0000 7f7e67fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:50.635 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.635+0000 7f7e67fff640 1 -- 172.21.15.6:0/3345558652 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1382104890 0 0) 0x7f7e801d4f80 con 0x7f7e800728f0 2024-03-25T19:14:50.635 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.635+0000 7f7e67fff640 10 monclient(hunting): _negotiate 2024-03-25T19:14:50.635 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.635+0000 7f7e67fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:50.635 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.635+0000 7f7e67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:50.635 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.635+0000 7f7e67fff640 10 monclient(hunting): my global_id is 4251 2024-03-25T19:14:50.635 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.635+0000 7f7e67fff640 1 -- 172.21.15.6:0/3345558652 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e6c004eb0 con 0x7f7e800728f0 2024-03-25T19:14:50.692 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.692+0000 7f7e67fff640 1 -- 172.21.15.6:0/3345558652 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1791331297 0 0) 0x7f7e8011ee40 con 0x7f7e8006f460 2024-03-25T19:14:50.692 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.692+0000 7f7e67fff640 10 monclient(hunting): _negotiate 2024-03-25T19:14:50.692 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.692+0000 7f7e67fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:50.692 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.692+0000 7f7e67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:50.692 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.692+0000 7f7e67fff640 10 monclient(hunting): my global_id is 4256 2024-03-25T19:14:50.692 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.692+0000 7f7e67fff640 1 -- 172.21.15.6:0/3345558652 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e801d4f80 con 0x7f7e8006f460 2024-03-25T19:14:50.710 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.710+0000 7f7e67fff640 1 -- 172.21.15.6:0/3345558652 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 848771023 0 0) 0x7f7e6c004eb0 con 0x7f7e800728f0 2024-03-25T19:14:50.710 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.710+0000 7f7e67fff640 1 -- 172.21.15.6:0/3345558652 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e8011ee40 con 0x7f7e800728f0 2024-03-25T19:14:50.710 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.710+0000 7f7e67fff640 1 -- 172.21.15.6:0/3345558652 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f7e78001b90 con 0x7f7e800728f0 2024-03-25T19:14:50.710 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.710+0000 7f7e67fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:50.710 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.710+0000 7f7e67fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:14:50.710 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.710+0000 7f7e67fff640 10 monclient(hunting): dump: 2024-03-25T19:14:50.710 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:14:50.711 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:50.711 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:50.711 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:50.711 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:50.711 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:50.711 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:50.711 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:50.711 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:50.711 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:50.711 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.710+0000 7f7e67fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:14:50.717 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.717+0000 7f7e67fff640 1 -- 172.21.15.6:0/3345558652 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2918414671 0 0) 0x7f7e801d4f80 con 0x7f7e8006f460 2024-03-25T19:14:50.717 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.717+0000 7f7e67fff640 1 -- 172.21.15.6:0/3345558652 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e6c004eb0 con 0x7f7e8006f460 2024-03-25T19:14:50.717 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.717+0000 7f7e67fff640 1 -- 172.21.15.6:0/3345558652 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f7e70001810 con 0x7f7e8006f460 2024-03-25T19:14:50.717 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.717+0000 7f7e67fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:50.717 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.717+0000 7f7e67fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:14:50.717 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.717+0000 7f7e67fff640 10 monclient(hunting): dump: 2024-03-25T19:14:50.717 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:14:50.717 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:50.717 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:50.718 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:50.718 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:50.718 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:50.718 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:50.718 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:50.718 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:50.718 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:50.718 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.718+0000 7f7e67fff640 1 -- 172.21.15.6:0/3345558652 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1918080802 0 0) 0x7f7e8011ebf0 con 0x7f7e8006f840 2024-03-25T19:14:50.718 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.718+0000 7f7e67fff640 10 monclient(hunting): _negotiate 2024-03-25T19:14:50.719 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.718+0000 7f7e67fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:50.719 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.718+0000 7f7e67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:50.719 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.718+0000 7f7e67fff640 10 monclient(hunting): my global_id is 4255 2024-03-25T19:14:50.719 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.718+0000 7f7e67fff640 1 -- 172.21.15.6:0/3345558652 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e801d4f80 con 0x7f7e8006f840 2024-03-25T19:14:50.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.721+0000 7f7e67fff640 1 -- 172.21.15.6:0/3345558652 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1424466655 0 0) 0x7f7e8011ee40 con 0x7f7e800728f0 2024-03-25T19:14:50.722 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.721+0000 7f7e67fff640 1 -- 172.21.15.6:0/3345558652 >> v1:172.21.15.6:6790/0 conn(0x7f7e8006f460 legacy=0x7f7e8011d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:50.722 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.721+0000 7f7e67fff640 1 -- 172.21.15.6:0/3345558652 >> v1:172.21.15.192:6789/0 conn(0x7f7e8006f840 legacy=0x7f7e8013ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:50.722 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.721+0000 7f7e67fff640 10 monclient: _finish_hunting 0 2024-03-25T19:14:50.722 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.721+0000 7f7e67fff640 1 monclient: found mon.a 2024-03-25T19:14:50.722 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.721+0000 7f7e67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:14:50.722 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.721+0000 7f7e67fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:14:50.722 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.721+0000 7f7e67fff640 1 -- 172.21.15.6:0/3345558652 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e800abfd0 con 0x7f7e800728f0 2024-03-25T19:14:50.722 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.721+0000 7f7e67fff640 10 monclient: _finish_auth 0 2024-03-25T19:14:50.722 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.721+0000 7f7e67fff640 10 monclient: _check_auth_tickets 2024-03-25T19:14:50.722 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.721+0000 7f7e67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:14:50.722 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.721+0000 7f7e85a3e640 5 monclient: authenticate success, global_id 4251 2024-03-25T19:14:50.722 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.721+0000 7f7e85a3e640 10 monclient: _renew_subs 2024-03-25T19:14:50.722 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.721+0000 7f7e85a3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:14:50.722 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.721+0000 7f7e85a3e640 1 -- 172.21.15.6:0/3345558652 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e8011dcc0 con 0x7f7e800728f0 2024-03-25T19:14:50.722 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.721+0000 7f7e85a3e640 10 client.4251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:14:50.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.721+0000 7f7e85a3e640 10 monclient: _renew_subs 2024-03-25T19:14:50.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.721+0000 7f7e85a3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:14:50.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.721+0000 7f7e85a3e640 1 -- 172.21.15.6:0/3345558652 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e8011c990 con 0x7f7e800728f0 2024-03-25T19:14:50.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.721+0000 7f7e85a3e640 1 librados: init done 2024-03-25T19:14:50.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.722+0000 7f7e85a3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:50.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.722+0000 7f7e85a3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:50.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.722+0000 7f7e85a3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:14:50.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:50.722+0000 7f7e85a3e640 1 -- 172.21.15.6:0/3345558652 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e80072d20 con 0x7f7e800728f0 2024-03-25T19:14:51.083 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:51.083+0000 7f7e67fff640 1 -- 172.21.15.6:0/3345558652 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e78002080 con 0x7f7e800728f0 2024-03-25T19:14:51.083 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:51.083+0000 7f7e67fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:14:51.083 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:51.083+0000 7f7e67fff640 1 -- 172.21.15.6:0/3345558652 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f7e78003b10 con 0x7f7e800728f0 2024-03-25T19:14:51.084 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:51.083+0000 7f7e67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:51.084 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:51.083+0000 7f7e67fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:14:51.084 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:51.083+0000 7f7e67fff640 10 monclient: dump: 2024-03-25T19:14:51.084 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:14:51.084 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:51.084 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:51.084 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:51.084 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:51.084 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:51.084 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:51.084 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:51.084 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:51.084 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:51.084 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:51.083+0000 7f7e7d7fa640 4 set_mon_vals no callback set 2024-03-25T19:14:51.085 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:51.084+0000 7f7e67fff640 1 -- 172.21.15.6:0/3345558652 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f7e78003d30 con 0x7f7e800728f0 2024-03-25T19:14:51.085 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:51.084+0000 7f7e7effd640 10 monclient: get_auth_request con 0x7f7e6c06df50 auth_method 0 2024-03-25T19:14:51.085 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:51.085+0000 7f7e67fff640 10 client.4251.objecter ms_handle_connect 0x7f7e6c06df50 2024-03-25T19:14:51.091 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:51.091+0000 7f7e67fff640 1 -- 172.21.15.6:0/3345558652 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f7e78053870 con 0x7f7e800728f0 2024-03-25T19:14:51.091 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:51.091+0000 7f7e67fff640 10 client.4251.objecter ms_dispatch 0x7f7e80000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:14:51.091 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:51.091+0000 7f7e67fff640 3 client.4251.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:14:51.091 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:51.091+0000 7f7e67fff640 3 client.4251.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:14:51.091 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:51.091+0000 7f7e67fff640 20 client.4251.objecter dump_active .. 0 homeless 2024-03-25T19:14:51.091 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:51.091+0000 7f7e67fff640 1 -- 172.21.15.6:0/3345558652 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7e7804f380 con 0x7f7e800728f0 2024-03-25T19:14:51.091 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:51.091+0000 7f7e67fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:51.091 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:51.091+0000 7f7e67fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:14:51.339 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:51.338+0000 7f7e85a3e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:14:51.339 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:51.338+0000 7f7e85a3e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:14:51.339 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:51.338+0000 7f7e85a3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:14:51.339 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:51.338+0000 7f7e85a3e640 1 -- 172.21.15.6:0/3345558652 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e80069f30 con 0x7f7e800728f0 2024-03-25T19:14:51.384 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:51.383+0000 7f7e67fff640 1 -- 172.21.15.6:0/3345558652 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f7e780512b0 con 0x7f7e800728f0 2024-03-25T19:14:51.384 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:51.383+0000 7f7e67fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:14:51.384 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:51.383+0000 7f7e67fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:14:51.384 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:14:51.384 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:14:51.387 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:51.386+0000 7f7e85a3e640 10 librados: watch_flush enter 2024-03-25T19:14:51.387 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:51.386+0000 7f7e85a3e640 10 librados: watch_flush exit 2024-03-25T19:14:51.387 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:51.386+0000 7f7e85a3e640 20 client.4251.objecter shutdown clearing up homeless session... 2024-03-25T19:14:51.387 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:51.386+0000 7f7e85a3e640 10 client.4251.objecter successfully canceled tick 2024-03-25T19:14:51.387 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:51.386+0000 7f7e85a3e640 1 -- 172.21.15.6:0/3345558652 >> v1:172.21.15.6:6800/2936652650 conn(0x7f7e6c06df50 legacy=0x7f7e6c070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:51.387 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:51.386+0000 7f7e85a3e640 10 monclient: shutdown 2024-03-25T19:14:51.387 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:51.386+0000 7f7e85a3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:14:51.387 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:51.386+0000 7f7e85a3e640 1 -- 172.21.15.6:0/3345558652 >> v1:172.21.15.6:6789/0 conn(0x7f7e800728f0 legacy=0x7f7e801d5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:51.387 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:51.386+0000 7f7e85a3e640 1 -- 172.21.15.6:0/3345558652 shutdown_connections 2024-03-25T19:14:51.387 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:51.386+0000 7f7e85a3e640 1 -- 172.21.15.6:0/3345558652 >> 172.21.15.6:0/3345558652 conn(0x7f7e800b4bd0 msgr2=0x7f7e8013c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:14:51.387 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:51.386+0000 7f7e85a3e640 1 -- 172.21.15.6:0/3345558652 shutdown_connections 2024-03-25T19:14:51.387 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:51.386+0000 7f7e85a3e640 1 -- 172.21.15.6:0/3345558652 wait complete. 2024-03-25T19:14:51.387 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:51.386+0000 7f7e85a3e640 1 librados: shutdown 2024-03-25T19:14:51.399 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:14:58.206 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:14:58.306 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.306+0000 7f240524a640 10 monclient: get_monmap_and_config 2024-03-25T19:14:58.306 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.306+0000 7f240524a640 10 monclient: build_initial_monmap 2024-03-25T19:14:58.306 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.306+0000 7f240524a640 10 monclient: monmap: 2024-03-25T19:14:58.307 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:14:58.307 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:58.307 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:14:58.307651+0000 2024-03-25T19:14:58.307 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:14:58.307651+0000 2024-03-25T19:14:58.307 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:14:58.307 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:58.307 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:14:58.307 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:14:58.307 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:14:58.307 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:58.308 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.307+0000 7f240524a640 1 Processor -- start 2024-03-25T19:14:58.308 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.307+0000 7f240524a640 1 -- start start 2024-03-25T19:14:58.308 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.307+0000 7f240524a640 10 monclient: init 2024-03-25T19:14:58.308 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.308+0000 7f240524a640 10 monclient: _reopen_session rank -1 2024-03-25T19:14:58.308 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.308+0000 7f240524a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:14:58.308 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.308+0000 7f240524a640 10 monclient(hunting): picked mon.noname-a con 0x7f24000b4e70 addr v1:172.21.15.6:6789/0 2024-03-25T19:14:58.309 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.308+0000 7f240524a640 10 monclient(hunting): picked mon.noname-c con 0x7f24000b44b0 addr v1:172.21.15.192:6789/0 2024-03-25T19:14:58.309 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.308+0000 7f240524a640 10 monclient(hunting): picked mon.noname-b con 0x7f24000b1b00 addr v1:172.21.15.6:6790/0 2024-03-25T19:14:58.309 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.308+0000 7f240524a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24000a96f0 con 0x7f24000b4e70 2024-03-25T19:14:58.309 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.308+0000 7f240524a640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24000ab0e0 con 0x7f24000b44b0 2024-03-25T19:14:58.309 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.308+0000 7f240524a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2400145f70 con 0x7f24000b1b00 2024-03-25T19:14:58.309 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.308+0000 7f240524a640 10 monclient(hunting): _renew_subs 2024-03-25T19:14:58.309 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.308+0000 7f240524a640 10 monclient(hunting): authenticate will time out at 6002.547970s 2024-03-25T19:14:58.314 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.314+0000 7f23fe59c640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f24000b44b0 0x7f2400145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:52388/0 (socket says 172.21.15.6:52388) 2024-03-25T19:14:58.314 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.314+0000 7f23fe59c640 1 -- 172.21.15.6:0/1438285733 learned_addr learned my addr 172.21.15.6:0/1438285733 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:14:58.332 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.332+0000 7f23fdd9b640 1 -- 172.21.15.6:0/1438285733 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 38793646 0 0) 0x7f24000ab0e0 con 0x7f24000b44b0 2024-03-25T19:14:58.332 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.332+0000 7f23fdd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:14:58.332 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.332+0000 7f23fdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:58.332 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.332+0000 7f23fdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:58.332 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.332+0000 7f23fdd9b640 10 monclient(hunting): my global_id is 4258 2024-03-25T19:14:58.332 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.332+0000 7f23fdd9b640 1 -- 172.21.15.6:0/1438285733 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23e8003520 con 0x7f24000b44b0 2024-03-25T19:14:58.336 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.336+0000 7f23fdd9b640 1 -- 172.21.15.6:0/1438285733 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4181394439 0 0) 0x7f2400145f70 con 0x7f24000b1b00 2024-03-25T19:14:58.336 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.336+0000 7f23fdd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:14:58.336 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.336+0000 7f23fdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:58.336 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.336+0000 7f23fdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:58.336 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.336+0000 7f23fdd9b640 10 monclient(hunting): my global_id is 4259 2024-03-25T19:14:58.336 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.336+0000 7f23fdd9b640 1 -- 172.21.15.6:0/1438285733 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24000ab0e0 con 0x7f24000b1b00 2024-03-25T19:14:58.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.341+0000 7f23fdd9b640 1 -- 172.21.15.6:0/1438285733 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3077222004 0 0) 0x7f24000ab0e0 con 0x7f24000b1b00 2024-03-25T19:14:58.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.341+0000 7f23fdd9b640 1 -- 172.21.15.6:0/1438285733 >> v1:172.21.15.192:6789/0 conn(0x7f24000b44b0 legacy=0x7f2400145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:58.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.341+0000 7f23fdd9b640 1 -- 172.21.15.6:0/1438285733 >> v1:172.21.15.6:6789/0 conn(0x7f24000b4e70 legacy=0x7f240013f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:58.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.341+0000 7f23fdd9b640 10 monclient: _finish_hunting 0 2024-03-25T19:14:58.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.341+0000 7f23fdd9b640 1 monclient: found mon.noname-b 2024-03-25T19:14:58.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.341+0000 7f23fdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:14:58.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.341+0000 7f23fdd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-03-25T19:14:58.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.341+0000 7f23fdd9b640 1 -- 172.21.15.6:0/1438285733 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24000a9b80 con 0x7f24000b1b00 2024-03-25T19:14:58.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.341+0000 7f23fdd9b640 10 monclient: _finish_auth 0 2024-03-25T19:14:58.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.341+0000 7f23fdd9b640 10 monclient: _check_auth_tickets 2024-03-25T19:14:58.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.341+0000 7f23fdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:14:58.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.341+0000 7f23fdd9b640 1 -- 172.21.15.6:0/1438285733 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f23f80019f0 con 0x7f24000b1b00 2024-03-25T19:14:58.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.341+0000 7f23fdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:58.343 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.341+0000 7f23fdd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:14:58.343 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.341+0000 7f23fdd9b640 10 monclient: dump: 2024-03-25T19:14:58.343 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:14:58.343 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:58.343 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:58.343 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:58.343 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:58.343 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:58.343 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:58.343 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:58.343 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:58.343 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:58.343 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.341+0000 7f240524a640 5 monclient: authenticate success, global_id 4259 2024-03-25T19:14:58.343 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.341+0000 7f240524a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:14:58.347 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.347+0000 7f23fdd9b640 1 -- 172.21.15.6:0/1438285733 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23f8001b50 con 0x7f24000b1b00 2024-03-25T19:14:58.347 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.347+0000 7f23fdd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:14:58.347 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.347+0000 7f23fdd9b640 1 -- 172.21.15.6:0/1438285733 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f23f8001fa0 con 0x7f24000b1b00 2024-03-25T19:14:58.347 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.347+0000 7f240524a640 10 monclient: get_monmap_and_config success 2024-03-25T19:14:58.347 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.347+0000 7f23fdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:58.347 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.347+0000 7f23fdd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:14:58.348 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.347+0000 7f23fdd9b640 10 monclient: dump: 2024-03-25T19:14:58.348 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:14:58.348 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:58.348 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:58.348 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:58.348 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:58.348 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:58.348 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:58.348 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:58.348 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:58.348 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:58.348 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.347+0000 7f240524a640 4 set_mon_vals no callback set 2024-03-25T19:14:58.350 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.348+0000 7f240524a640 10 monclient: shutdown 2024-03-25T19:14:58.350 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.348+0000 7f240524a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:14:58.350 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.348+0000 7f240524a640 1 -- 172.21.15.6:0/1438285733 >> v1:172.21.15.6:6790/0 conn(0x7f24000b1b00 legacy=0x7f240014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:58.350 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.348+0000 7f240524a640 1 -- 172.21.15.6:0/1438285733 shutdown_connections 2024-03-25T19:14:58.350 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.348+0000 7f240524a640 1 -- 172.21.15.6:0/1438285733 >> 172.21.15.6:0/1438285733 conn(0x7f24000617a0 msgr2=0x7f2400061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:14:58.350 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.348+0000 7f240524a640 1 -- 172.21.15.6:0/1438285733 shutdown_connections 2024-03-25T19:14:58.350 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.348+0000 7f240524a640 1 -- 172.21.15.6:0/1438285733 wait complete. 2024-03-25T19:14:58.350 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.349+0000 7f240524a640 10 monclient: build_initial_monmap 2024-03-25T19:14:58.350 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.349+0000 7f240524a640 10 monclient: monmap: 2024-03-25T19:14:58.350 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:14:58.351 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:14:58.351 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:14:58.351 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:14:58.351 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:14:58.351 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:14:58.351 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:14:58.351 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:14:58.351 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:14:58.351 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:58.351 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.349+0000 7f240524a640 1 librados: starting msgr at 2024-03-25T19:14:58.351 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.349+0000 7f240524a640 1 librados: starting objecter 2024-03-25T19:14:58.351 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.350+0000 7f240524a640 1 Processor -- start 2024-03-25T19:14:58.351 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.350+0000 7f240524a640 1 -- start start 2024-03-25T19:14:58.351 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.350+0000 7f240524a640 1 librados: setting wanted keys 2024-03-25T19:14:58.351 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.350+0000 7f240524a640 1 librados: calling monclient init 2024-03-25T19:14:58.352 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.350+0000 7f240524a640 10 monclient: init 2024-03-25T19:14:58.352 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.350+0000 7f240524a640 10 monclient: _reopen_session rank -1 2024-03-25T19:14:58.352 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.350+0000 7f240524a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:14:58.352 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.350+0000 7f240524a640 10 monclient(hunting): picked mon.noname-c con 0x7f24000b1b00 addr v1:172.21.15.6:6790/0 2024-03-25T19:14:58.352 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.350+0000 7f240524a640 10 monclient(hunting): picked mon.noname-b con 0x7f24000b44b0 addr v1:172.21.15.192:6789/0 2024-03-25T19:14:58.353 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.350+0000 7f240524a640 10 monclient(hunting): picked mon.noname-a con 0x7f24000b4e70 addr v1:172.21.15.6:6789/0 2024-03-25T19:14:58.353 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.350+0000 7f240524a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f240012b0b0 con 0x7f24000b4e70 2024-03-25T19:14:58.353 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.350+0000 7f240524a640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f240012b390 con 0x7f24000b44b0 2024-03-25T19:14:58.353 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.351+0000 7f240524a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f240013bfc0 con 0x7f24000b1b00 2024-03-25T19:14:58.353 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.351+0000 7f240524a640 10 monclient(hunting): _renew_subs 2024-03-25T19:14:58.353 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.351+0000 7f240524a640 10 monclient(hunting): authenticate will time out at 6002.590789s 2024-03-25T19:14:58.365 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.366+0000 7f23fed9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f24000b1b00 0x7f240012b7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:32840/0 (socket says 172.21.15.6:32840) 2024-03-25T19:14:58.366 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.366+0000 7f23fed9d640 1 -- 172.21.15.6:0/2439393923 learned_addr learned my addr 172.21.15.6:0/2439393923 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:14:58.370 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.370+0000 7f23e77fe640 10 client.?.objecter ms_handle_connect 0x7f24000b1b00 2024-03-25T19:14:58.370 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.370+0000 7f23e77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:58.371 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.371+0000 7f23e77fe640 10 client.?.objecter ms_handle_connect 0x7f24000b44b0 2024-03-25T19:14:58.371 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.371+0000 7f23e77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:58.381 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.380+0000 7f23e77fe640 10 client.?.objecter ms_handle_connect 0x7f24000b4e70 2024-03-25T19:14:58.381 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.380+0000 7f23e77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:14:58.381 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.380+0000 7f23e77fe640 1 -- 172.21.15.6:0/2439393923 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3956869975 0 0) 0x7f240012b390 con 0x7f24000b44b0 2024-03-25T19:14:58.381 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.380+0000 7f23e77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:14:58.381 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.380+0000 7f23e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:58.381 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.380+0000 7f23e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:58.381 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.380+0000 7f23e77fe640 10 monclient(hunting): my global_id is 4261 2024-03-25T19:14:58.381 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.380+0000 7f23e77fe640 1 -- 172.21.15.6:0/2439393923 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23ec004eb0 con 0x7f24000b44b0 2024-03-25T19:14:58.390 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.390+0000 7f23e77fe640 1 -- 172.21.15.6:0/2439393923 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4155868618 0 0) 0x7f240013bfc0 con 0x7f24000b1b00 2024-03-25T19:14:58.390 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.390+0000 7f23e77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:14:58.390 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.390+0000 7f23e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:58.390 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.390+0000 7f23e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:58.390 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.390+0000 7f23e77fe640 10 monclient(hunting): my global_id is 4262 2024-03-25T19:14:58.390 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.390+0000 7f23e77fe640 1 -- 172.21.15.6:0/2439393923 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f240012b390 con 0x7f24000b1b00 2024-03-25T19:14:58.400 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.400+0000 7f23e77fe640 1 -- 172.21.15.6:0/2439393923 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 795928443 0 0) 0x7f23ec004eb0 con 0x7f24000b44b0 2024-03-25T19:14:58.400 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.400+0000 7f23e77fe640 1 -- 172.21.15.6:0/2439393923 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f240013bfc0 con 0x7f24000b44b0 2024-03-25T19:14:58.401 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.400+0000 7f23e77fe640 1 -- 172.21.15.6:0/2439393923 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f23f4001950 con 0x7f24000b44b0 2024-03-25T19:14:58.401 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.400+0000 7f23e77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:58.401 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.400+0000 7f23e77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:14:58.401 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.400+0000 7f23e77fe640 10 monclient(hunting): dump: 2024-03-25T19:14:58.401 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:14:58.401 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:58.401 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:58.402 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:58.402 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:58.402 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:58.402 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:58.402 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:58.402 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:58.402 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:58.403 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.400+0000 7f23e77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:14:58.404 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.404+0000 7f23e77fe640 1 -- 172.21.15.6:0/2439393923 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1488128254 0 0) 0x7f240012b0b0 con 0x7f24000b4e70 2024-03-25T19:14:58.404 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.404+0000 7f23e77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:14:58.404 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.404+0000 7f23e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:14:58.404 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.404+0000 7f23e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:14:58.404 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.404+0000 7f23e77fe640 10 monclient(hunting): my global_id is 4257 2024-03-25T19:14:58.404 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.404+0000 7f23e77fe640 1 -- 172.21.15.6:0/2439393923 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23ec004eb0 con 0x7f24000b4e70 2024-03-25T19:14:58.415 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.414+0000 7f23e77fe640 1 -- 172.21.15.6:0/2439393923 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1400261550 0 0) 0x7f240013bfc0 con 0x7f24000b44b0 2024-03-25T19:14:58.415 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.414+0000 7f23e77fe640 1 -- 172.21.15.6:0/2439393923 >> v1:172.21.15.6:6790/0 conn(0x7f24000b1b00 legacy=0x7f240012b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:58.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.414+0000 7f23e77fe640 1 -- 172.21.15.6:0/2439393923 >> v1:172.21.15.6:6789/0 conn(0x7f24000b4e70 legacy=0x7f24001e1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:58.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.414+0000 7f23e77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:14:58.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.414+0000 7f23e77fe640 1 monclient: found mon.b 2024-03-25T19:14:58.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.414+0000 7f23e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:14:58.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.414+0000 7f23e77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:58.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.414+0000 7f23e77fe640 1 -- 172.21.15.6:0/2439393923 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f240013bbe0 con 0x7f24000b44b0 2024-03-25T19:14:58.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.414+0000 7f23e77fe640 10 monclient: _finish_auth 0 2024-03-25T19:14:58.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.414+0000 7f23e77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:14:58.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.414+0000 7f23e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:14:58.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.414+0000 7f240524a640 5 monclient: authenticate success, global_id 4261 2024-03-25T19:14:58.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.414+0000 7f240524a640 10 monclient: _renew_subs 2024-03-25T19:14:58.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.414+0000 7f240524a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:58.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.414+0000 7f240524a640 1 -- 172.21.15.6:0/2439393923 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f240013c4a0 con 0x7f24000b44b0 2024-03-25T19:14:58.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.414+0000 7f240524a640 10 client.4261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:14:58.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.414+0000 7f240524a640 10 monclient: _renew_subs 2024-03-25T19:14:58.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.414+0000 7f240524a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:58.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.414+0000 7f240524a640 1 -- 172.21.15.6:0/2439393923 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2400139a20 con 0x7f24000b44b0 2024-03-25T19:14:58.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.414+0000 7f240524a640 1 librados: init done 2024-03-25T19:14:58.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.415+0000 7f240524a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:58.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.415+0000 7f240524a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:58.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.415+0000 7f240524a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:58.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.415+0000 7f240524a640 1 -- 172.21.15.6:0/2439393923 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f240013be30 con 0x7f24000b44b0 2024-03-25T19:14:58.701 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.701+0000 7f23e77fe640 1 -- 172.21.15.6:0/2439393923 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23f4001d60 con 0x7f24000b44b0 2024-03-25T19:14:58.701 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.701+0000 7f23e77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:14:58.701 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.701+0000 7f23e77fe640 1 -- 172.21.15.6:0/2439393923 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f23f4004980 con 0x7f24000b44b0 2024-03-25T19:14:58.701 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.701+0000 7f23e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:14:58.701 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.701+0000 7f23e77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:14:58.702 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.701+0000 7f23e77fe640 10 monclient: dump: 2024-03-25T19:14:58.702 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:14:58.702 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:14:58.702 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:58.702 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:14:58.702 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:14:58.702 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:14:58.702 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:14:58.702 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:14:58.702 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:14:58.702 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:14:58.702 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.701+0000 7f23fd59a640 4 set_mon_vals no callback set 2024-03-25T19:14:58.703 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.702+0000 7f23e77fe640 1 -- 172.21.15.6:0/2439393923 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f23f4004ba0 con 0x7f24000b44b0 2024-03-25T19:14:58.703 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.702+0000 7f23fed9d640 10 monclient: get_auth_request con 0x7f23ec06ddb0 auth_method 0 2024-03-25T19:14:58.703 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.702+0000 7f23e77fe640 10 client.4261.objecter ms_handle_connect 0x7f23ec06ddb0 2024-03-25T19:14:58.708 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.708+0000 7f23e77fe640 1 -- 172.21.15.6:0/2439393923 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f23f4054730 con 0x7f24000b44b0 2024-03-25T19:14:58.708 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.708+0000 7f23e77fe640 10 client.4261.objecter ms_dispatch 0x7f2400000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:14:58.708 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.708+0000 7f23e77fe640 3 client.4261.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:14:58.708 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.708+0000 7f23e77fe640 3 client.4261.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:14:58.708 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.708+0000 7f23e77fe640 20 client.4261.objecter dump_active .. 0 homeless 2024-03-25T19:14:58.709 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.709+0000 7f23e77fe640 1 -- 172.21.15.6:0/2439393923 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f23f4050070 con 0x7f24000b44b0 2024-03-25T19:14:58.709 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.709+0000 7f23e77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:14:58.709 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.709+0000 7f23e77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:14:58.964 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.963+0000 7f240524a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:14:58.964 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.963+0000 7f240524a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:14:58.964 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.963+0000 7f240524a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:14:58.964 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.963+0000 7f240524a640 1 -- 172.21.15.6:0/2439393923 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f240013c150 con 0x7f24000b44b0 2024-03-25T19:14:58.992 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.992+0000 7f23e77fe640 1 -- 172.21.15.6:0/2439393923 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f23f4052170 con 0x7f24000b44b0 2024-03-25T19:14:58.992 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.992+0000 7f23e77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:14:58.992 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.992+0000 7f23e77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:14:58.992 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:14:58.992 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:14:58.995 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.994+0000 7f240524a640 10 librados: watch_flush enter 2024-03-25T19:14:58.995 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.994+0000 7f240524a640 10 librados: watch_flush exit 2024-03-25T19:14:58.995 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.994+0000 7f240524a640 20 client.4261.objecter shutdown clearing up homeless session... 2024-03-25T19:14:58.995 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.994+0000 7f240524a640 10 client.4261.objecter successfully canceled tick 2024-03-25T19:14:58.995 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.995+0000 7f240524a640 1 -- 172.21.15.6:0/2439393923 >> v1:172.21.15.6:6800/2936652650 conn(0x7f23ec06ddb0 legacy=0x7f23ec070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:58.996 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.995+0000 7f240524a640 10 monclient: shutdown 2024-03-25T19:14:58.996 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.995+0000 7f240524a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:14:58.996 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.995+0000 7f240524a640 1 -- 172.21.15.6:0/2439393923 >> v1:172.21.15.192:6789/0 conn(0x7f24000b44b0 legacy=0x7f240013e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:14:58.996 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.995+0000 7f240524a640 1 -- 172.21.15.6:0/2439393923 shutdown_connections 2024-03-25T19:14:58.996 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.995+0000 7f240524a640 1 -- 172.21.15.6:0/2439393923 >> 172.21.15.6:0/2439393923 conn(0x7f24000617a0 msgr2=0x7f24000b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:14:58.996 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.995+0000 7f240524a640 1 -- 172.21.15.6:0/2439393923 shutdown_connections 2024-03-25T19:14:58.996 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.995+0000 7f240524a640 1 -- 172.21.15.6:0/2439393923 wait complete. 2024-03-25T19:14:58.996 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:14:58.995+0000 7f240524a640 1 librados: shutdown 2024-03-25T19:14:59.007 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:15:02.077 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:01:19.424 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:15:02.135 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:01:19.482 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:15:02.221 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:01:19.538 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:15:02.230 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:01:19.576 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:15:02.281 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:01:19.599 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:15:02.304 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:01:19.651 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:15:02.402 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:01:19.749 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:15:02.496 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:01:19.814 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:15:02.568 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:01:19.914 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:15:05.812 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:15:05.927 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.927+0000 7f86655e8640 10 monclient: get_monmap_and_config 2024-03-25T19:15:05.928 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.927+0000 7f86655e8640 10 monclient: build_initial_monmap 2024-03-25T19:15:05.928 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.927+0000 7f86655e8640 10 monclient: monmap: 2024-03-25T19:15:05.928 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:15:05.928 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:05.928 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:15:05.928786+0000 2024-03-25T19:15:05.928 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:15:05.928786+0000 2024-03-25T19:15:05.928 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:15:05.928 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:05.928 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:15:05.928 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:15:05.928 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:15:05.928 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:05.930 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.928+0000 7f86655e8640 1 Processor -- start 2024-03-25T19:15:05.930 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.928+0000 7f86655e8640 1 -- start start 2024-03-25T19:15:05.930 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.928+0000 7f86655e8640 10 monclient: init 2024-03-25T19:15:05.930 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.929+0000 7f86655e8640 10 monclient: _reopen_session rank -1 2024-03-25T19:15:05.930 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.929+0000 7f86655e8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:15:05.930 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.929+0000 7f86655e8640 10 monclient(hunting): picked mon.noname-a con 0x7f86600b4ff0 addr v1:172.21.15.6:6789/0 2024-03-25T19:15:05.930 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.929+0000 7f86655e8640 10 monclient(hunting): picked mon.noname-c con 0x7f86600b4670 addr v1:172.21.15.192:6789/0 2024-03-25T19:15:05.931 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.929+0000 7f86655e8640 10 monclient(hunting): picked mon.noname-b con 0x7f866013c480 addr v1:172.21.15.6:6790/0 2024-03-25T19:15:05.931 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.929+0000 7f86655e8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86600ab0e0 con 0x7f86600b4ff0 2024-03-25T19:15:05.931 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.929+0000 7f86655e8640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86600ab3c0 con 0x7f86600b4670 2024-03-25T19:15:05.931 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.930+0000 7f86655e8640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86600a9f30 con 0x7f866013c480 2024-03-25T19:15:05.931 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.930+0000 7f86655e8640 10 monclient(hunting): _renew_subs 2024-03-25T19:15:05.931 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.930+0000 7f86655e8640 10 monclient(hunting): authenticate will time out at 6010.169924s 2024-03-25T19:15:05.941 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.941+0000 7f865effd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f86600b4ff0 0x7f866013bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50158/0 (socket says 172.21.15.6:50158) 2024-03-25T19:15:05.941 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.941+0000 7f865effd640 1 -- 172.21.15.6:0/3210970671 learned_addr learned my addr 172.21.15.6:0/3210970671 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:15:05.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.958+0000 7f865dffb640 1 -- 172.21.15.6:0/3210970671 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 290767833 0 0) 0x7f86600ab0e0 con 0x7f86600b4ff0 2024-03-25T19:15:05.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.958+0000 7f865dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:15:05.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.958+0000 7f865dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:15:05.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.958+0000 7f865dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:15:05.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.958+0000 7f865dffb640 10 monclient(hunting): my global_id is 4260 2024-03-25T19:15:05.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.958+0000 7f865dffb640 1 -- 172.21.15.6:0/3210970671 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8648003520 con 0x7f86600b4ff0 2024-03-25T19:15:05.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.958+0000 7f865dffb640 1 -- 172.21.15.6:0/3210970671 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 39038559 0 0) 0x7f86600a9f30 con 0x7f866013c480 2024-03-25T19:15:05.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.958+0000 7f865dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:15:05.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.958+0000 7f865dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:15:05.959 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.958+0000 7f865dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:15:05.959 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.958+0000 7f865dffb640 10 monclient(hunting): my global_id is 4265 2024-03-25T19:15:05.959 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.958+0000 7f865dffb640 1 -- 172.21.15.6:0/3210970671 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86600ab0e0 con 0x7f866013c480 2024-03-25T19:15:05.966 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.966+0000 7f865dffb640 1 -- 172.21.15.6:0/3210970671 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3859078960 0 0) 0x7f86600ab3c0 con 0x7f86600b4670 2024-03-25T19:15:05.966 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.966+0000 7f865dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:15:05.966 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.966+0000 7f865dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:15:05.966 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.966+0000 7f865dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:15:05.967 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.966+0000 7f865dffb640 10 monclient(hunting): my global_id is 4264 2024-03-25T19:15:05.967 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.966+0000 7f865dffb640 1 -- 172.21.15.6:0/3210970671 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86600a9f30 con 0x7f86600b4670 2024-03-25T19:15:05.971 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.970+0000 7f865dffb640 1 -- 172.21.15.6:0/3210970671 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3567157664 0 0) 0x7f86600ab0e0 con 0x7f866013c480 2024-03-25T19:15:05.971 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.970+0000 7f865dffb640 1 -- 172.21.15.6:0/3210970671 >> v1:172.21.15.192:6789/0 conn(0x7f86600b4670 legacy=0x7f86601425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:05.971 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.970+0000 7f865dffb640 1 -- 172.21.15.6:0/3210970671 >> v1:172.21.15.6:6789/0 conn(0x7f86600b4ff0 legacy=0x7f866013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:05.971 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.970+0000 7f865dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:15:05.971 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.970+0000 7f865dffb640 1 monclient: found mon.noname-b 2024-03-25T19:15:05.971 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.970+0000 7f865dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:15:05.971 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.970+0000 7f865dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-03-25T19:15:05.971 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.970+0000 7f865dffb640 1 -- 172.21.15.6:0/3210970671 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86600a9b70 con 0x7f866013c480 2024-03-25T19:15:05.971 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.970+0000 7f865dffb640 10 monclient: _finish_auth 0 2024-03-25T19:15:05.971 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.970+0000 7f865dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:15:05.971 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.970+0000 7f865dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:15:05.971 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.970+0000 7f865dffb640 1 -- 172.21.15.6:0/3210970671 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f86580019f0 con 0x7f866013c480 2024-03-25T19:15:05.972 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.970+0000 7f86655e8640 5 monclient: authenticate success, global_id 4265 2024-03-25T19:15:05.972 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.970+0000 7f86655e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:15:05.972 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.970+0000 7f865dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:15:05.972 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.970+0000 7f865dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:15:05.972 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.970+0000 7f865dffb640 10 monclient: dump: 2024-03-25T19:15:05.972 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:15:05.972 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:05.972 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:05.972 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:05.972 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:15:05.972 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:05.972 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:15:05.972 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:15:05.972 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:15:05.972 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:05.972 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.970+0000 7f86655e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:15:05.976 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.976+0000 7f865dffb640 1 -- 172.21.15.6:0/3210970671 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8658001b50 con 0x7f866013c480 2024-03-25T19:15:05.977 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.976+0000 7f865dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:15:05.977 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.976+0000 7f865dffb640 1 -- 172.21.15.6:0/3210970671 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f8658001f60 con 0x7f866013c480 2024-03-25T19:15:05.977 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.976+0000 7f86655e8640 10 monclient: get_monmap_and_config success 2024-03-25T19:15:05.977 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.976+0000 7f86655e8640 4 set_mon_vals no callback set 2024-03-25T19:15:05.977 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.976+0000 7f865dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:15:05.977 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.976+0000 7f865dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:15:05.977 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.976+0000 7f865dffb640 10 monclient: dump: 2024-03-25T19:15:05.977 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:15:05.977 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:05.977 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:05.977 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:05.977 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:15:05.977 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:05.977 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:15:05.977 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:15:05.978 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:15:05.978 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:05.978 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.977+0000 7f86655e8640 10 monclient: shutdown 2024-03-25T19:15:05.979 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.977+0000 7f86655e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:15:05.979 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.977+0000 7f86655e8640 1 -- 172.21.15.6:0/3210970671 >> v1:172.21.15.6:6790/0 conn(0x7f866013c480 legacy=0x7f866014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:05.979 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.978+0000 7f86655e8640 1 -- 172.21.15.6:0/3210970671 shutdown_connections 2024-03-25T19:15:05.979 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.978+0000 7f86655e8640 1 -- 172.21.15.6:0/3210970671 >> 172.21.15.6:0/3210970671 conn(0x7f86600617a0 msgr2=0x7f8660061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:15:05.979 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.978+0000 7f86655e8640 1 -- 172.21.15.6:0/3210970671 shutdown_connections 2024-03-25T19:15:05.979 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.978+0000 7f86655e8640 1 -- 172.21.15.6:0/3210970671 wait complete. 2024-03-25T19:15:05.979 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.979+0000 7f86655e8640 10 monclient: build_initial_monmap 2024-03-25T19:15:05.979 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.979+0000 7f86655e8640 10 monclient: monmap: 2024-03-25T19:15:05.979 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:15:05.979 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:15:05.979 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:15:05.979 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:15:05.979 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:15:05.980 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:15:05.980 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:15:05.980 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:15:05.980 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:15:05.980 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:05.981 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.979+0000 7f86655e8640 1 librados: starting msgr at 2024-03-25T19:15:05.981 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.979+0000 7f86655e8640 1 librados: starting objecter 2024-03-25T19:15:05.981 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.979+0000 7f86655e8640 1 Processor -- start 2024-03-25T19:15:05.981 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.979+0000 7f86655e8640 1 -- start start 2024-03-25T19:15:05.981 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.979+0000 7f86655e8640 1 librados: setting wanted keys 2024-03-25T19:15:05.981 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.979+0000 7f86655e8640 1 librados: calling monclient init 2024-03-25T19:15:05.981 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.979+0000 7f86655e8640 10 monclient: init 2024-03-25T19:15:05.982 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.980+0000 7f86655e8640 10 monclient: _reopen_session rank -1 2024-03-25T19:15:05.982 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.980+0000 7f86655e8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:15:05.982 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.980+0000 7f86655e8640 10 monclient(hunting): picked mon.noname-c con 0x7f86600b4670 addr v1:172.21.15.6:6790/0 2024-03-25T19:15:05.982 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.980+0000 7f86655e8640 10 monclient(hunting): picked mon.noname-a con 0x7f86600b4ff0 addr v1:172.21.15.6:6789/0 2024-03-25T19:15:05.982 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.980+0000 7f86655e8640 10 monclient(hunting): picked mon.noname-b con 0x7f866013c480 addr v1:172.21.15.192:6789/0 2024-03-25T19:15:05.982 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.980+0000 7f86655e8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f866013f640 con 0x7f86600b4ff0 2024-03-25T19:15:05.982 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.980+0000 7f86655e8640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8660145010 con 0x7f866013c480 2024-03-25T19:15:05.982 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.980+0000 7f86655e8640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f866013f1d0 con 0x7f86600b4670 2024-03-25T19:15:05.982 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.980+0000 7f86655e8640 10 monclient(hunting): _renew_subs 2024-03-25T19:15:05.982 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.980+0000 7f86655e8640 10 monclient(hunting): authenticate will time out at 6010.220737s 2024-03-25T19:15:05.992 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.992+0000 7f865e7fc640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f86600b4ff0 0x7f86601303c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50194/0 (socket says 172.21.15.6:50194) 2024-03-25T19:15:05.992 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.992+0000 7f865e7fc640 1 -- 172.21.15.6:0/280289092 learned_addr learned my addr 172.21.15.6:0/280289092 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:15:05.992 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.993+0000 7f86477fe640 10 client.?.objecter ms_handle_connect 0x7f86600b4670 2024-03-25T19:15:05.992 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.993+0000 7f86477fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:15:05.993 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.993+0000 7f86477fe640 10 client.?.objecter ms_handle_connect 0x7f86600b4ff0 2024-03-25T19:15:05.993 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:05.994+0000 7f86477fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:15:06.000 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.000+0000 7f86477fe640 1 -- 172.21.15.6:0/280289092 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3239026110 0 0) 0x7f866013f1d0 con 0x7f86600b4670 2024-03-25T19:15:06.000 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.000+0000 7f86477fe640 10 monclient(hunting): _negotiate 2024-03-25T19:15:06.000 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.000+0000 7f86477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:15:06.000 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.000+0000 7f86477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:15:06.000 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.000+0000 7f86477fe640 10 monclient(hunting): my global_id is 4268 2024-03-25T19:15:06.000 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.000+0000 7f86477fe640 1 -- 172.21.15.6:0/280289092 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f864c004eb0 con 0x7f86600b4670 2024-03-25T19:15:06.000 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.000+0000 7f86477fe640 1 -- 172.21.15.6:0/280289092 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1352481977 0 0) 0x7f866013f640 con 0x7f86600b4ff0 2024-03-25T19:15:06.000 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.000+0000 7f86477fe640 10 monclient(hunting): _negotiate 2024-03-25T19:15:06.000 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.000+0000 7f86477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:15:06.000 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.000+0000 7f86477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:15:06.000 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.000+0000 7f86477fe640 10 monclient(hunting): my global_id is 4263 2024-03-25T19:15:06.001 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.000+0000 7f86477fe640 1 -- 172.21.15.6:0/280289092 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f866013f1d0 con 0x7f86600b4ff0 2024-03-25T19:15:06.008 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.007+0000 7f86477fe640 1 -- 172.21.15.6:0/280289092 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1061609269 0 0) 0x7f866013f1d0 con 0x7f86600b4ff0 2024-03-25T19:15:06.008 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.007+0000 7f86477fe640 1 -- 172.21.15.6:0/280289092 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f866013f640 con 0x7f86600b4ff0 2024-03-25T19:15:06.008 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.007+0000 7f86477fe640 1 -- 172.21.15.6:0/280289092 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f86540019f0 con 0x7f86600b4ff0 2024-03-25T19:15:06.008 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.007+0000 7f86477fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:15:06.008 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.008+0000 7f86477fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:15:06.008 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.008+0000 7f86477fe640 10 monclient(hunting): dump: 2024-03-25T19:15:06.008 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:15:06.008 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:06.008 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:06.009 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:06.009 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:15:06.009 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:06.009 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:15:06.009 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:15:06.009 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:15:06.009 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:06.009 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.008+0000 7f86477fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:15:06.009 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.008+0000 7f86477fe640 1 -- 172.21.15.6:0/280289092 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2116550785 0 0) 0x7f864c004eb0 con 0x7f86600b4670 2024-03-25T19:15:06.009 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.008+0000 7f86477fe640 1 -- 172.21.15.6:0/280289092 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f866013f1d0 con 0x7f86600b4670 2024-03-25T19:15:06.009 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.008+0000 7f86477fe640 1 -- 172.21.15.6:0/280289092 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f86500018d0 con 0x7f86600b4670 2024-03-25T19:15:06.009 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.008+0000 7f86477fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:15:06.009 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.008+0000 7f86477fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:15:06.009 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.008+0000 7f86477fe640 10 monclient(hunting): dump: 2024-03-25T19:15:06.010 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:15:06.010 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:06.010 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:06.010 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:06.010 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:15:06.010 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:06.010 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:15:06.010 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:15:06.010 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:15:06.010 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:06.012 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.012+0000 7f86477fe640 10 client.?.objecter ms_handle_connect 0x7f866013c480 2024-03-25T19:15:06.012 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.012+0000 7f86477fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:15:06.021 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.021+0000 7f86477fe640 1 -- 172.21.15.6:0/280289092 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1319533601 0 0) 0x7f866013f640 con 0x7f86600b4ff0 2024-03-25T19:15:06.021 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.021+0000 7f86477fe640 1 -- 172.21.15.6:0/280289092 >> v1:172.21.15.6:6790/0 conn(0x7f86600b4670 legacy=0x7f86601dbed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:06.021 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.021+0000 7f86477fe640 1 -- 172.21.15.6:0/280289092 >> v1:172.21.15.192:6789/0 conn(0x7f866013c480 legacy=0x7f86601e3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:06.021 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.021+0000 7f86477fe640 10 monclient: _finish_hunting 0 2024-03-25T19:15:06.021 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.021+0000 7f86477fe640 1 monclient: found mon.a 2024-03-25T19:15:06.021 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.021+0000 7f86477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:15:06.021 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.021+0000 7f86477fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:15:06.021 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.021+0000 7f86477fe640 1 -- 172.21.15.6:0/280289092 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f866013f360 con 0x7f86600b4ff0 2024-03-25T19:15:06.021 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.021+0000 7f86477fe640 10 monclient: _finish_auth 0 2024-03-25T19:15:06.022 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.021+0000 7f86477fe640 10 monclient: _check_auth_tickets 2024-03-25T19:15:06.022 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.021+0000 7f86477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:15:06.022 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.021+0000 7f86655e8640 5 monclient: authenticate success, global_id 4263 2024-03-25T19:15:06.022 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.021+0000 7f86655e8640 10 monclient: _renew_subs 2024-03-25T19:15:06.022 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.021+0000 7f86655e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:15:06.022 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.021+0000 7f86655e8640 1 -- 172.21.15.6:0/280289092 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8660130ad0 con 0x7f86600b4ff0 2024-03-25T19:15:06.022 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.021+0000 7f86655e8640 10 client.4263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:15:06.022 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.021+0000 7f86655e8640 10 monclient: _renew_subs 2024-03-25T19:15:06.022 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.021+0000 7f86655e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:15:06.022 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.021+0000 7f86655e8640 1 -- 172.21.15.6:0/280289092 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8660143840 con 0x7f86600b4ff0 2024-03-25T19:15:06.022 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.021+0000 7f86655e8640 1 librados: init done 2024-03-25T19:15:06.023 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.022+0000 7f86655e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:15:06.023 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.022+0000 7f86655e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:15:06.023 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.022+0000 7f86655e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:15:06.023 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.022+0000 7f86655e8640 1 -- 172.21.15.6:0/280289092 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86600b3ff0 con 0x7f86600b4ff0 2024-03-25T19:15:06.345 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.345+0000 7f86477fe640 1 -- 172.21.15.6:0/280289092 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8654003610 con 0x7f86600b4ff0 2024-03-25T19:15:06.345 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.345+0000 7f86477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:15:06.345 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.345+0000 7f86477fe640 1 -- 172.21.15.6:0/280289092 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f86540039e0 con 0x7f86600b4ff0 2024-03-25T19:15:06.345 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.345+0000 7f86477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:15:06.345 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.345+0000 7f86477fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:15:06.345 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.345+0000 7f86477fe640 10 monclient: dump: 2024-03-25T19:15:06.345 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:15:06.345 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:06.345 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:06.345 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:06.346 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:15:06.346 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:06.346 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:15:06.346 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:15:06.346 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:15:06.346 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:06.346 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.345+0000 7f865cff9640 4 set_mon_vals no callback set 2024-03-25T19:15:06.346 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.346+0000 7f86477fe640 1 -- 172.21.15.6:0/280289092 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f8654003c00 con 0x7f86600b4ff0 2024-03-25T19:15:06.347 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.346+0000 7f865effd640 10 monclient: get_auth_request con 0x7f864c06de90 auth_method 0 2024-03-25T19:15:06.347 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.346+0000 7f86477fe640 10 client.4263.objecter ms_handle_connect 0x7f864c06de90 2024-03-25T19:15:06.352 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.352+0000 7f86477fe640 1 -- 172.21.15.6:0/280289092 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f8654054840 con 0x7f86600b4ff0 2024-03-25T19:15:06.352 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.352+0000 7f86477fe640 10 client.4263.objecter ms_dispatch 0x7f8660000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:15:06.352 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.352+0000 7f86477fe640 3 client.4263.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:15:06.352 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.352+0000 7f86477fe640 3 client.4263.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:15:06.353 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.352+0000 7f86477fe640 20 client.4263.objecter dump_active .. 0 homeless 2024-03-25T19:15:06.353 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.352+0000 7f86477fe640 1 -- 172.21.15.6:0/280289092 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8654050350 con 0x7f86600b4ff0 2024-03-25T19:15:06.353 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.352+0000 7f86477fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:15:06.353 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.352+0000 7f86477fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:15:06.571 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.571+0000 7f86655e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:15:06.572 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.571+0000 7f86655e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:15:06.572 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.571+0000 7f86655e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:15:06.572 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.571+0000 7f86655e8640 1 -- 172.21.15.6:0/280289092 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f866006dbd0 con 0x7f86600b4ff0 2024-03-25T19:15:06.604 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.604+0000 7f86477fe640 1 -- 172.21.15.6:0/280289092 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f8654052280 con 0x7f86600b4ff0 2024-03-25T19:15:06.604 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.604+0000 7f86477fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:15:06.604 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.604+0000 7f86477fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:15:06.605 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:15:06.605 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:15:06.607 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.606+0000 7f86655e8640 10 librados: watch_flush enter 2024-03-25T19:15:06.607 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.607+0000 7f86655e8640 10 librados: watch_flush exit 2024-03-25T19:15:06.607 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.607+0000 7f86655e8640 20 client.4263.objecter shutdown clearing up homeless session... 2024-03-25T19:15:06.608 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.607+0000 7f86655e8640 10 client.4263.objecter successfully canceled tick 2024-03-25T19:15:06.608 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.607+0000 7f86655e8640 1 -- 172.21.15.6:0/280289092 >> v1:172.21.15.6:6800/2936652650 conn(0x7f864c06de90 legacy=0x7f864c070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:06.608 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.607+0000 7f86655e8640 10 monclient: shutdown 2024-03-25T19:15:06.608 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.607+0000 7f86655e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:15:06.608 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.607+0000 7f86655e8640 1 -- 172.21.15.6:0/280289092 >> v1:172.21.15.6:6789/0 conn(0x7f86600b4ff0 legacy=0x7f86601303c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:06.608 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.607+0000 7f86655e8640 1 -- 172.21.15.6:0/280289092 shutdown_connections 2024-03-25T19:15:06.608 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.607+0000 7f86655e8640 1 -- 172.21.15.6:0/280289092 >> 172.21.15.6:0/280289092 conn(0x7f86600617a0 msgr2=0x7f86600b2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:15:06.608 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.607+0000 7f86655e8640 1 -- 172.21.15.6:0/280289092 shutdown_connections 2024-03-25T19:15:06.608 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.607+0000 7f86655e8640 1 -- 172.21.15.6:0/280289092 wait complete. 2024-03-25T19:15:06.608 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:06.607+0000 7f86655e8640 1 librados: shutdown 2024-03-25T19:15:06.619 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:15:13.425 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:15:13.543 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.543+0000 7fb7a6f4b640 10 monclient: get_monmap_and_config 2024-03-25T19:15:13.543 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.543+0000 7fb7a6f4b640 10 monclient: build_initial_monmap 2024-03-25T19:15:13.543 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.543+0000 7fb7a6f4b640 10 monclient: monmap: 2024-03-25T19:15:13.543 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:15:13.543 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:13.543 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:15:13.544426+0000 2024-03-25T19:15:13.543 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:15:13.544426+0000 2024-03-25T19:15:13.543 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:15:13.544 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:13.544 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:15:13.544 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:15:13.544 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:15:13.544 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:13.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.544+0000 7fb7a6f4b640 1 Processor -- start 2024-03-25T19:15:13.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.544+0000 7fb7a6f4b640 1 -- start start 2024-03-25T19:15:13.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.544+0000 7fb7a6f4b640 10 monclient: init 2024-03-25T19:15:13.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.544+0000 7fb7a6f4b640 10 monclient: _reopen_session rank -1 2024-03-25T19:15:13.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.544+0000 7fb7a6f4b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:15:13.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.544+0000 7fb7a6f4b640 10 monclient(hunting): picked mon.noname-a con 0x7fb7a00b4e70 addr v1:172.21.15.6:6789/0 2024-03-25T19:15:13.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.545+0000 7fb7a6f4b640 10 monclient(hunting): picked mon.noname-c con 0x7fb7a00b44b0 addr v1:172.21.15.192:6789/0 2024-03-25T19:15:13.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.545+0000 7fb7a6f4b640 10 monclient(hunting): picked mon.noname-b con 0x7fb7a00b1b00 addr v1:172.21.15.6:6790/0 2024-03-25T19:15:13.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.545+0000 7fb7a6f4b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7a00a96f0 con 0x7fb7a00b4e70 2024-03-25T19:15:13.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.545+0000 7fb7a6f4b640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7a00ab0e0 con 0x7fb7a00b44b0 2024-03-25T19:15:13.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.545+0000 7fb7a6f4b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7a0145f70 con 0x7fb7a00b1b00 2024-03-25T19:15:13.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.545+0000 7fb7a6f4b640 10 monclient(hunting): _renew_subs 2024-03-25T19:15:13.546 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.545+0000 7fb7a6f4b640 10 monclient(hunting): authenticate will time out at 6017.784886s 2024-03-25T19:15:13.553 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.554+0000 7fb797fff640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fb7a00b44b0 0x7fb7a0145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:46074/0 (socket says 172.21.15.6:46074) 2024-03-25T19:15:13.554 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.554+0000 7fb797fff640 1 -- 172.21.15.6:0/1972111790 learned_addr learned my addr 172.21.15.6:0/1972111790 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:15:13.570 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.570+0000 7fb7977fe640 1 -- 172.21.15.6:0/1972111790 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3359858620 0 0) 0x7fb7a00ab0e0 con 0x7fb7a00b44b0 2024-03-25T19:15:13.570 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.570+0000 7fb7977fe640 10 monclient(hunting): _negotiate 2024-03-25T19:15:13.571 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.570+0000 7fb7977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:15:13.571 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.570+0000 7fb7977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:15:13.571 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.570+0000 7fb7977fe640 10 monclient(hunting): my global_id is 4270 2024-03-25T19:15:13.571 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.570+0000 7fb7977fe640 1 -- 172.21.15.6:0/1972111790 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb788003520 con 0x7fb7a00b44b0 2024-03-25T19:15:13.582 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.582+0000 7fb7977fe640 1 -- 172.21.15.6:0/1972111790 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3433374032 0 0) 0x7fb788003520 con 0x7fb7a00b44b0 2024-03-25T19:15:13.582 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.582+0000 7fb7977fe640 1 -- 172.21.15.6:0/1972111790 >> v1:172.21.15.6:6790/0 conn(0x7fb7a00b1b00 legacy=0x7fb7a014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:13.582 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.582+0000 7fb7977fe640 1 -- 172.21.15.6:0/1972111790 >> v1:172.21.15.6:6789/0 conn(0x7fb7a00b4e70 legacy=0x7fb7a013f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:13.583 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.582+0000 7fb7977fe640 10 monclient: _finish_hunting 0 2024-03-25T19:15:13.583 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.582+0000 7fb7977fe640 1 monclient: found mon.noname-c 2024-03-25T19:15:13.583 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.582+0000 7fb7977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:15:13.583 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.582+0000 7fb7977fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:15:13.583 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.582+0000 7fb7977fe640 1 -- 172.21.15.6:0/1972111790 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a00a9b80 con 0x7fb7a00b44b0 2024-03-25T19:15:13.583 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.582+0000 7fb7977fe640 10 monclient: _finish_auth 0 2024-03-25T19:15:13.583 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.582+0000 7fb7977fe640 10 monclient: _check_auth_tickets 2024-03-25T19:15:13.583 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.582+0000 7fb7977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:15:13.583 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.582+0000 7fb7977fe640 1 -- 172.21.15.6:0/1972111790 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fb790001690 con 0x7fb7a00b44b0 2024-03-25T19:15:13.583 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.582+0000 7fb7977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:15:13.583 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.582+0000 7fb7977fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:15:13.583 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.582+0000 7fb7977fe640 10 monclient: dump: 2024-03-25T19:15:13.583 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:15:13.583 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:13.584 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:13.584 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:13.584 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:15:13.584 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:13.584 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:15:13.584 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:15:13.584 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:15:13.584 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:13.584 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.582+0000 7fb7a6f4b640 5 monclient: authenticate success, global_id 4270 2024-03-25T19:15:13.584 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.582+0000 7fb7a6f4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:15:13.591 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.591+0000 7fb7977fe640 1 -- 172.21.15.6:0/1972111790 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7900017f0 con 0x7fb7a00b44b0 2024-03-25T19:15:13.591 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.591+0000 7fb7977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:15:13.591 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.591+0000 7fb7977fe640 1 -- 172.21.15.6:0/1972111790 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fb790001c40 con 0x7fb7a00b44b0 2024-03-25T19:15:13.592 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.591+0000 7fb7a6f4b640 10 monclient: get_monmap_and_config success 2024-03-25T19:15:13.592 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.591+0000 7fb7a6f4b640 4 set_mon_vals no callback set 2024-03-25T19:15:13.592 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.591+0000 7fb7977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:15:13.592 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.591+0000 7fb7977fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:15:13.592 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.591+0000 7fb7977fe640 10 monclient: dump: 2024-03-25T19:15:13.592 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:15:13.592 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:13.592 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:13.592 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:13.592 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:15:13.592 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:13.592 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:15:13.592 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:15:13.592 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:15:13.592 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:13.594 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.592+0000 7fb7a6f4b640 10 monclient: shutdown 2024-03-25T19:15:13.594 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.592+0000 7fb7a6f4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:15:13.594 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.592+0000 7fb7a6f4b640 1 -- 172.21.15.6:0/1972111790 >> v1:172.21.15.192:6789/0 conn(0x7fb7a00b44b0 legacy=0x7fb7a0145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:13.594 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.592+0000 7fb7a6f4b640 1 -- 172.21.15.6:0/1972111790 shutdown_connections 2024-03-25T19:15:13.594 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.592+0000 7fb7a6f4b640 1 -- 172.21.15.6:0/1972111790 >> 172.21.15.6:0/1972111790 conn(0x7fb7a00617a0 msgr2=0x7fb7a0061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:15:13.594 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.592+0000 7fb7a6f4b640 1 -- 172.21.15.6:0/1972111790 shutdown_connections 2024-03-25T19:15:13.594 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.592+0000 7fb7a6f4b640 1 -- 172.21.15.6:0/1972111790 wait complete. 2024-03-25T19:15:13.594 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.593+0000 7fb7a6f4b640 10 monclient: build_initial_monmap 2024-03-25T19:15:13.594 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.593+0000 7fb7a6f4b640 10 monclient: monmap: 2024-03-25T19:15:13.594 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:15:13.595 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:15:13.595 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:15:13.595 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:15:13.595 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:15:13.595 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:15:13.595 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:15:13.595 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:15:13.595 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:15:13.595 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:13.595 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.594+0000 7fb7a6f4b640 1 librados: starting msgr at 2024-03-25T19:15:13.595 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.594+0000 7fb7a6f4b640 1 librados: starting objecter 2024-03-25T19:15:13.595 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.594+0000 7fb7a6f4b640 1 Processor -- start 2024-03-25T19:15:13.595 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.594+0000 7fb7a6f4b640 1 -- start start 2024-03-25T19:15:13.595 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.594+0000 7fb7a6f4b640 1 librados: setting wanted keys 2024-03-25T19:15:13.595 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.594+0000 7fb7a6f4b640 1 librados: calling monclient init 2024-03-25T19:15:13.596 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.594+0000 7fb7a6f4b640 10 monclient: init 2024-03-25T19:15:13.597 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.595+0000 7fb7a6f4b640 10 monclient: _reopen_session rank -1 2024-03-25T19:15:13.597 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.595+0000 7fb7a6f4b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:15:13.597 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.595+0000 7fb7a6f4b640 10 monclient(hunting): picked mon.noname-b con 0x7fb7a00b1b00 addr v1:172.21.15.192:6789/0 2024-03-25T19:15:13.597 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.595+0000 7fb7a6f4b640 10 monclient(hunting): picked mon.noname-c con 0x7fb7a00b44b0 addr v1:172.21.15.6:6790/0 2024-03-25T19:15:13.597 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.595+0000 7fb7a6f4b640 10 monclient(hunting): picked mon.noname-a con 0x7fb7a00b4e70 addr v1:172.21.15.6:6789/0 2024-03-25T19:15:13.597 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.595+0000 7fb7a6f4b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7a0137f60 con 0x7fb7a00b4e70 2024-03-25T19:15:13.597 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.595+0000 7fb7a6f4b640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7a01369e0 con 0x7fb7a00b1b00 2024-03-25T19:15:13.597 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.595+0000 7fb7a6f4b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7a0139bd0 con 0x7fb7a00b44b0 2024-03-25T19:15:13.597 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.595+0000 7fb7a6f4b640 10 monclient(hunting): _renew_subs 2024-03-25T19:15:13.597 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.595+0000 7fb7a6f4b640 10 monclient(hunting): authenticate will time out at 6017.835478s 2024-03-25T19:15:13.602 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.602+0000 7fb797fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fb7a00b44b0 0x7fb7a01dac50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:58382/0 (socket says 172.21.15.6:58382) 2024-03-25T19:15:13.602 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.602+0000 7fb797fff640 1 -- 172.21.15.6:0/1523740792 learned_addr learned my addr 172.21.15.6:0/1523740792 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:15:13.604 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.604+0000 7fb7957fa640 10 client.?.objecter ms_handle_connect 0x7fb7a00b44b0 2024-03-25T19:15:13.604 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.604+0000 7fb7957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:15:13.608 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.608+0000 7fb7957fa640 10 client.?.objecter ms_handle_connect 0x7fb7a00b1b00 2024-03-25T19:15:13.608 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.608+0000 7fb7957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:15:13.613 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.613+0000 7fb7957fa640 1 -- 172.21.15.6:0/1523740792 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3465350213 0 0) 0x7fb7a0139bd0 con 0x7fb7a00b44b0 2024-03-25T19:15:13.613 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.613+0000 7fb7957fa640 10 monclient(hunting): _negotiate 2024-03-25T19:15:13.614 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.613+0000 7fb7957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:15:13.614 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.613+0000 7fb7957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:15:13.614 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.613+0000 7fb7957fa640 10 monclient(hunting): my global_id is 4274 2024-03-25T19:15:13.614 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.613+0000 7fb7957fa640 1 -- 172.21.15.6:0/1523740792 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb78c004eb0 con 0x7fb7a00b44b0 2024-03-25T19:15:13.618 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.618+0000 7fb7957fa640 10 client.?.objecter ms_handle_connect 0x7fb7a00b4e70 2024-03-25T19:15:13.618 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.618+0000 7fb7957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:15:13.618 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.618+0000 7fb7957fa640 1 -- 172.21.15.6:0/1523740792 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1308307814 0 0) 0x7fb7a01369e0 con 0x7fb7a00b1b00 2024-03-25T19:15:13.618 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.618+0000 7fb7957fa640 10 monclient(hunting): _negotiate 2024-03-25T19:15:13.618 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.618+0000 7fb7957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:15:13.618 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.618+0000 7fb7957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:15:13.618 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.618+0000 7fb7957fa640 10 monclient(hunting): my global_id is 4273 2024-03-25T19:15:13.618 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.618+0000 7fb7957fa640 1 -- 172.21.15.6:0/1523740792 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7a0139bd0 con 0x7fb7a00b1b00 2024-03-25T19:15:13.629 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.629+0000 7fb7957fa640 1 -- 172.21.15.6:0/1523740792 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3651879715 0 0) 0x7fb78c004eb0 con 0x7fb7a00b44b0 2024-03-25T19:15:13.629 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.629+0000 7fb7957fa640 1 -- 172.21.15.6:0/1523740792 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7a01369e0 con 0x7fb7a00b44b0 2024-03-25T19:15:13.629 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.629+0000 7fb7957fa640 1 -- 172.21.15.6:0/1523740792 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fb790001b40 con 0x7fb7a00b44b0 2024-03-25T19:15:13.629 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.629+0000 7fb7957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:15:13.630 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.629+0000 7fb7957fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:15:13.630 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.629+0000 7fb7957fa640 10 monclient(hunting): dump: 2024-03-25T19:15:13.630 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:15:13.630 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:13.630 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:13.630 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:13.630 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:15:13.630 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:13.630 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:15:13.630 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:15:13.630 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:15:13.630 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:13.631 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.629+0000 7fb7957fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:15:13.633 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.633+0000 7fb7957fa640 1 -- 172.21.15.6:0/1523740792 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 70351697 0 0) 0x7fb7a0139bd0 con 0x7fb7a00b1b00 2024-03-25T19:15:13.633 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.633+0000 7fb7957fa640 1 -- 172.21.15.6:0/1523740792 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb78c004eb0 con 0x7fb7a00b1b00 2024-03-25T19:15:13.633 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.633+0000 7fb7957fa640 1 -- 172.21.15.6:0/1523740792 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fb79c001810 con 0x7fb7a00b1b00 2024-03-25T19:15:13.633 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.633+0000 7fb7957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:15:13.633 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.633+0000 7fb7957fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:15:13.633 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.633+0000 7fb7957fa640 10 monclient(hunting): dump: 2024-03-25T19:15:13.633 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:15:13.633 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:13.633 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:13.634 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:13.634 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:15:13.634 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:13.634 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:15:13.634 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:15:13.634 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:15:13.634 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:13.641 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.641+0000 7fb7957fa640 1 -- 172.21.15.6:0/1523740792 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 507792950 0 0) 0x7fb7a0137f60 con 0x7fb7a00b4e70 2024-03-25T19:15:13.641 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.641+0000 7fb7957fa640 10 monclient(hunting): _negotiate 2024-03-25T19:15:13.641 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.641+0000 7fb7957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:15:13.641 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.641+0000 7fb7957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:15:13.641 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.641+0000 7fb7957fa640 10 monclient(hunting): my global_id is 4269 2024-03-25T19:15:13.641 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.641+0000 7fb7957fa640 1 -- 172.21.15.6:0/1523740792 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7a0139bd0 con 0x7fb7a00b4e70 2024-03-25T19:15:13.649 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.649+0000 7fb7957fa640 1 -- 172.21.15.6:0/1523740792 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3950500335 0 0) 0x7fb7a01369e0 con 0x7fb7a00b44b0 2024-03-25T19:15:13.649 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.649+0000 7fb7957fa640 1 -- 172.21.15.6:0/1523740792 >> v1:172.21.15.192:6789/0 conn(0x7fb7a00b1b00 legacy=0x7fb7a0135ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:13.649 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.649+0000 7fb7957fa640 1 -- 172.21.15.6:0/1523740792 >> v1:172.21.15.6:6789/0 conn(0x7fb7a00b4e70 legacy=0x7fb7a01db360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:13.650 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.649+0000 7fb7957fa640 10 monclient: _finish_hunting 0 2024-03-25T19:15:13.650 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.649+0000 7fb7957fa640 1 monclient: found mon.c 2024-03-25T19:15:13.650 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.649+0000 7fb7957fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:15:13.650 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.649+0000 7fb7957fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:15:13.650 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.649+0000 7fb7957fa640 1 -- 172.21.15.6:0/1523740792 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a0139690 con 0x7fb7a00b44b0 2024-03-25T19:15:13.650 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.649+0000 7fb7957fa640 10 monclient: _finish_auth 0 2024-03-25T19:15:13.650 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.649+0000 7fb7957fa640 10 monclient: _check_auth_tickets 2024-03-25T19:15:13.650 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.649+0000 7fb7957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:15:13.650 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.649+0000 7fb7a6f4b640 5 monclient: authenticate success, global_id 4274 2024-03-25T19:15:13.650 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.649+0000 7fb7a6f4b640 10 monclient: _renew_subs 2024-03-25T19:15:13.650 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.649+0000 7fb7a6f4b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:15:13.650 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.649+0000 7fb7a6f4b640 1 -- 172.21.15.6:0/1523740792 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7a0139940 con 0x7fb7a00b44b0 2024-03-25T19:15:13.650 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.649+0000 7fb7a6f4b640 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:15:13.651 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.649+0000 7fb7a6f4b640 10 monclient: _renew_subs 2024-03-25T19:15:13.651 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.649+0000 7fb7a6f4b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:15:13.651 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.649+0000 7fb7a6f4b640 1 -- 172.21.15.6:0/1523740792 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7a01e3380 con 0x7fb7a00b44b0 2024-03-25T19:15:13.651 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.649+0000 7fb7a6f4b640 1 librados: init done 2024-03-25T19:15:13.651 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.650+0000 7fb7a6f4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:15:13.651 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.650+0000 7fb7a6f4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:15:13.651 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.650+0000 7fb7a6f4b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:15:13.651 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.650+0000 7fb7a6f4b640 1 -- 172.21.15.6:0/1523740792 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7a01367c0 con 0x7fb7a00b44b0 2024-03-25T19:15:13.963 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.963+0000 7fb7957fa640 1 -- 172.21.15.6:0/1523740792 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb790004bf0 con 0x7fb7a00b44b0 2024-03-25T19:15:13.964 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.963+0000 7fb7957fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:15:13.964 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.963+0000 7fb7957fa640 1 -- 172.21.15.6:0/1523740792 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fb790001d80 con 0x7fb7a00b44b0 2024-03-25T19:15:13.964 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.963+0000 7fb7957fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:15:13.964 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.963+0000 7fb7957fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:15:13.964 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.963+0000 7fb7957fa640 10 monclient: dump: 2024-03-25T19:15:13.964 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:15:13.964 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:13.964 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:13.964 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:13.964 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:15:13.964 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:13.964 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:15:13.964 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:15:13.964 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:15:13.965 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:13.965 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.963+0000 7fb796ffd640 4 set_mon_vals no callback set 2024-03-25T19:15:13.965 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.964+0000 7fb7957fa640 1 -- 172.21.15.6:0/1523740792 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7fb79001ad70 con 0x7fb7a00b44b0 2024-03-25T19:15:13.965 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.965+0000 7fb7a4ce7640 10 monclient: get_auth_request con 0x7fb78c06deb0 auth_method 0 2024-03-25T19:15:13.965 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.965+0000 7fb7957fa640 10 client.4274.objecter ms_handle_connect 0x7fb78c06deb0 2024-03-25T19:15:13.970 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.970+0000 7fb7957fa640 1 -- 172.21.15.6:0/1523740792 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7fb7900549a0 con 0x7fb7a00b44b0 2024-03-25T19:15:13.970 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.970+0000 7fb7957fa640 10 client.4274.objecter ms_dispatch 0x7fb7a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:15:13.971 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.970+0000 7fb7957fa640 3 client.4274.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:15:13.971 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.970+0000 7fb7957fa640 3 client.4274.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:15:13.971 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.970+0000 7fb7957fa640 20 client.4274.objecter dump_active .. 0 homeless 2024-03-25T19:15:13.971 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.970+0000 7fb7957fa640 1 -- 172.21.15.6:0/1523740792 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb7900502e0 con 0x7fb7a00b44b0 2024-03-25T19:15:13.971 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.970+0000 7fb7957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:15:13.971 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:13.970+0000 7fb7957fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:15:14.195 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:14.195+0000 7fb7a6f4b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:15:14.195 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:14.195+0000 7fb7a6f4b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:15:14.195 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:14.195+0000 7fb7a6f4b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:15:14.195 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:14.195+0000 7fb7a6f4b640 1 -- 172.21.15.6:0/1523740792 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7a004f730 con 0x7fb7a00b44b0 2024-03-25T19:15:14.217 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:14.217+0000 7fb7957fa640 1 -- 172.21.15.6:0/1523740792 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7fb7900523e0 con 0x7fb7a00b44b0 2024-03-25T19:15:14.217 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:14.217+0000 7fb7957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:15:14.217 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:14.217+0000 7fb7957fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:15:14.217 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:15:14.217 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:15:14.220 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:14.219+0000 7fb7a6f4b640 10 librados: watch_flush enter 2024-03-25T19:15:14.220 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:14.219+0000 7fb7a6f4b640 10 librados: watch_flush exit 2024-03-25T19:15:14.220 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:14.219+0000 7fb7a6f4b640 20 client.4274.objecter shutdown clearing up homeless session... 2024-03-25T19:15:14.220 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:14.219+0000 7fb7a6f4b640 10 client.4274.objecter successfully canceled tick 2024-03-25T19:15:14.220 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:14.219+0000 7fb7a6f4b640 1 -- 172.21.15.6:0/1523740792 >> v1:172.21.15.6:6800/2936652650 conn(0x7fb78c06deb0 legacy=0x7fb78c070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:14.220 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:14.219+0000 7fb7a6f4b640 10 monclient: shutdown 2024-03-25T19:15:14.220 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:14.219+0000 7fb7a6f4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:15:14.221 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:14.219+0000 7fb7a6f4b640 1 -- 172.21.15.6:0/1523740792 >> v1:172.21.15.6:6790/0 conn(0x7fb7a00b44b0 legacy=0x7fb7a01dac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:14.221 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:14.220+0000 7fb7a6f4b640 1 -- 172.21.15.6:0/1523740792 shutdown_connections 2024-03-25T19:15:14.221 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:14.220+0000 7fb7a6f4b640 1 -- 172.21.15.6:0/1523740792 >> 172.21.15.6:0/1523740792 conn(0x7fb7a00617a0 msgr2=0x7fb7a00b6600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:15:14.221 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:14.220+0000 7fb7a6f4b640 1 -- 172.21.15.6:0/1523740792 shutdown_connections 2024-03-25T19:15:14.221 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:14.220+0000 7fb7a6f4b640 1 -- 172.21.15.6:0/1523740792 wait complete. 2024-03-25T19:15:14.221 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:14.220+0000 7fb7a6f4b640 1 librados: shutdown 2024-03-25T19:15:14.231 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:15:21.037 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:15:21.156 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.156+0000 7f8c27d13640 10 monclient: get_monmap_and_config 2024-03-25T19:15:21.156 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.156+0000 7f8c27d13640 10 monclient: build_initial_monmap 2024-03-25T19:15:21.156 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.156+0000 7f8c27d13640 10 monclient: monmap: 2024-03-25T19:15:21.156 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:15:21.156 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:21.156 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:15:21.157285+0000 2024-03-25T19:15:21.156 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:15:21.157285+0000 2024-03-25T19:15:21.156 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:15:21.157 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:21.157 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:15:21.157 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:15:21.157 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:15:21.157 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:21.158 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.157+0000 7f8c27d13640 1 Processor -- start 2024-03-25T19:15:21.158 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.157+0000 7f8c27d13640 1 -- start start 2024-03-25T19:15:21.158 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.157+0000 7f8c27d13640 10 monclient: init 2024-03-25T19:15:21.158 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.158+0000 7f8c27d13640 10 monclient: _reopen_session rank -1 2024-03-25T19:15:21.158 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.158+0000 7f8c27d13640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:15:21.158 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.158+0000 7f8c27d13640 10 monclient(hunting): picked mon.noname-b con 0x7f8c200b4ff0 addr v1:172.21.15.6:6790/0 2024-03-25T19:15:21.159 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.158+0000 7f8c27d13640 10 monclient(hunting): picked mon.noname-c con 0x7f8c200b4670 addr v1:172.21.15.192:6789/0 2024-03-25T19:15:21.159 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.158+0000 7f8c27d13640 10 monclient(hunting): picked mon.noname-a con 0x7f8c2013c480 addr v1:172.21.15.6:6789/0 2024-03-25T19:15:21.159 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.158+0000 7f8c27d13640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c200ab0e0 con 0x7f8c2013c480 2024-03-25T19:15:21.159 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.158+0000 7f8c27d13640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c200ab3c0 con 0x7f8c200b4670 2024-03-25T19:15:21.159 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.158+0000 7f8c27d13640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c200a9f30 con 0x7f8c200b4ff0 2024-03-25T19:15:21.159 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.158+0000 7f8c27d13640 10 monclient(hunting): _renew_subs 2024-03-25T19:15:21.159 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.158+0000 7f8c27d13640 10 monclient(hunting): authenticate will time out at 6025.398029s 2024-03-25T19:15:21.167 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.167+0000 7f8c252ae640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f8c200b4670 0x7f8c201425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:32846/0 (socket says 172.21.15.6:32846) 2024-03-25T19:15:21.167 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.167+0000 7f8c252ae640 1 -- 172.21.15.6:0/1026837504 learned_addr learned my addr 172.21.15.6:0/1026837504 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:15:21.190 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.189+0000 7f8c24aad640 1 -- 172.21.15.6:0/1026837504 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1795165755 0 0) 0x7f8c200ab0e0 con 0x7f8c2013c480 2024-03-25T19:15:21.190 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.189+0000 7f8c24aad640 10 monclient(hunting): _negotiate 2024-03-25T19:15:21.190 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.189+0000 7f8c24aad640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:15:21.190 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.189+0000 7f8c24aad640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:15:21.190 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.189+0000 7f8c24aad640 10 monclient(hunting): my global_id is 4272 2024-03-25T19:15:21.190 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.189+0000 7f8c24aad640 1 -- 172.21.15.6:0/1026837504 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c0c003520 con 0x7f8c2013c480 2024-03-25T19:15:21.190 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.190+0000 7f8c24aad640 1 -- 172.21.15.6:0/1026837504 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1675903123 0 0) 0x7f8c200ab3c0 con 0x7f8c200b4670 2024-03-25T19:15:21.190 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.190+0000 7f8c24aad640 10 monclient(hunting): _negotiate 2024-03-25T19:15:21.190 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.190+0000 7f8c24aad640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:15:21.190 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.190+0000 7f8c24aad640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:15:21.190 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.190+0000 7f8c24aad640 10 monclient(hunting): my global_id is 4276 2024-03-25T19:15:21.191 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.190+0000 7f8c24aad640 1 -- 172.21.15.6:0/1026837504 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c200ab0e0 con 0x7f8c200b4670 2024-03-25T19:15:21.194 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.194+0000 7f8c252ae640 1 -- 172.21.15.6:0/1026837504 >> v1:172.21.15.192:6789/0 conn(0x7f8c200b4670 legacy=0x7f8c201425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-03-25T19:15:21.194 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.194+0000 7f8c252ae640 1 -- 172.21.15.6:0/1026837504 >> v1:172.21.15.192:6789/0 conn(0x7f8c200b4670 legacy=0x7f8c201425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:15:21.194 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.194+0000 7f8c252ae640 1 --1- 172.21.15.6:0/1026837504 >> v1:172.21.15.192:6789/0 conn(0x7f8c200b4670 0x7f8c201425c0 :-1 s=OPENED pgs=4670 cs=1 l=1).handle_message read tag failed 2024-03-25T19:15:21.194 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.194+0000 7f8c252ae640 1 --1- 172.21.15.6:0/1026837504 >> v1:172.21.15.192:6789/0 conn(0x7f8c200b4670 0x7f8c201425c0 :-1 s=OPENED pgs=4670 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:15:21.194 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.194+0000 7f8c24aad640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.192:6789/0 2024-03-25T19:15:21.205 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.205+0000 7f8c24aad640 1 -- 172.21.15.6:0/1026837504 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1358099344 0 0) 0x7f8c200a9f30 con 0x7f8c200b4ff0 2024-03-25T19:15:21.206 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.205+0000 7f8c24aad640 10 monclient(hunting): _negotiate 2024-03-25T19:15:21.206 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.205+0000 7f8c24aad640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:15:21.206 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.205+0000 7f8c24aad640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:15:21.206 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.205+0000 7f8c24aad640 10 monclient(hunting): my global_id is 4277 2024-03-25T19:15:21.206 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.205+0000 7f8c24aad640 1 -- 172.21.15.6:0/1026837504 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c200ab3c0 con 0x7f8c200b4ff0 2024-03-25T19:15:21.210 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.210+0000 7f8c24aad640 1 -- 172.21.15.6:0/1026837504 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3553670066 0 0) 0x7f8c0c003520 con 0x7f8c2013c480 2024-03-25T19:15:21.211 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.210+0000 7f8c24aad640 1 -- 172.21.15.6:0/1026837504 >> v1:172.21.15.6:6790/0 conn(0x7f8c200b4ff0 legacy=0x7f8c2013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:21.211 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.210+0000 7f8c24aad640 1 -- 172.21.15.6:0/1026837504 >> v1:172.21.15.192:6789/0 conn(0x7f8c200b4670 legacy=0x7f8c201425c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:15:21.211 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.210+0000 7f8c24aad640 10 monclient: _finish_hunting 0 2024-03-25T19:15:21.211 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.210+0000 7f8c24aad640 1 monclient: found mon.noname-a 2024-03-25T19:15:21.211 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.210+0000 7f8c24aad640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:15:21.211 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.210+0000 7f8c24aad640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-03-25T19:15:21.211 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.210+0000 7f8c24aad640 1 -- 172.21.15.6:0/1026837504 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c200a9b70 con 0x7f8c2013c480 2024-03-25T19:15:21.211 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.210+0000 7f8c24aad640 10 monclient: _finish_auth 0 2024-03-25T19:15:21.211 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.210+0000 7f8c24aad640 10 monclient: _check_auth_tickets 2024-03-25T19:15:21.211 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.210+0000 7f8c24aad640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:15:21.211 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.210+0000 7f8c24aad640 1 -- 172.21.15.6:0/1026837504 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f8c18001a10 con 0x7f8c2013c480 2024-03-25T19:15:21.211 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.210+0000 7f8c24aad640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:15:21.211 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.210+0000 7f8c24aad640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:15:21.211 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.210+0000 7f8c24aad640 10 monclient: dump: 2024-03-25T19:15:21.212 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:15:21.212 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:21.212 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:21.212 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:21.212 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:15:21.212 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:21.212 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:15:21.212 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:15:21.212 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:15:21.212 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:21.212 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.210+0000 7f8c27d13640 5 monclient: authenticate success, global_id 4272 2024-03-25T19:15:21.212 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.210+0000 7f8c27d13640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:15:21.223 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.223+0000 7f8c24aad640 1 -- 172.21.15.6:0/1026837504 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c18001b70 con 0x7f8c2013c480 2024-03-25T19:15:21.223 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.223+0000 7f8c24aad640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:15:21.223 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.223+0000 7f8c24aad640 1 -- 172.21.15.6:0/1026837504 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f8c18001f80 con 0x7f8c2013c480 2024-03-25T19:15:21.223 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.223+0000 7f8c27d13640 10 monclient: get_monmap_and_config success 2024-03-25T19:15:21.223 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.223+0000 7f8c27d13640 4 set_mon_vals no callback set 2024-03-25T19:15:21.224 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.223+0000 7f8c24aad640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:15:21.224 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.223+0000 7f8c24aad640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:15:21.224 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.223+0000 7f8c24aad640 10 monclient: dump: 2024-03-25T19:15:21.224 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:15:21.224 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:21.224 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:21.224 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:21.224 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:15:21.224 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:21.224 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:15:21.224 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:15:21.224 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:15:21.224 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:21.226 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.224+0000 7f8c27d13640 10 monclient: shutdown 2024-03-25T19:15:21.226 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.224+0000 7f8c27d13640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:15:21.226 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.224+0000 7f8c27d13640 1 -- 172.21.15.6:0/1026837504 >> v1:172.21.15.6:6789/0 conn(0x7f8c2013c480 legacy=0x7f8c2014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:21.226 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.224+0000 7f8c27d13640 1 -- 172.21.15.6:0/1026837504 shutdown_connections 2024-03-25T19:15:21.226 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.224+0000 7f8c27d13640 1 -- 172.21.15.6:0/1026837504 >> 172.21.15.6:0/1026837504 conn(0x7f8c200617a0 msgr2=0x7f8c20061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:15:21.226 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.224+0000 7f8c27d13640 1 -- 172.21.15.6:0/1026837504 shutdown_connections 2024-03-25T19:15:21.226 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.224+0000 7f8c27d13640 1 -- 172.21.15.6:0/1026837504 wait complete. 2024-03-25T19:15:21.226 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.225+0000 7f8c27d13640 10 monclient: build_initial_monmap 2024-03-25T19:15:21.227 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.225+0000 7f8c27d13640 10 monclient: monmap: 2024-03-25T19:15:21.227 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:15:21.227 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:15:21.227 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:15:21.227 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:15:21.227 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:15:21.227 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:15:21.227 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:15:21.227 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:15:21.227 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:15:21.227 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:21.227 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.225+0000 7f8c27d13640 1 librados: starting msgr at 2024-03-25T19:15:21.227 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.225+0000 7f8c27d13640 1 librados: starting objecter 2024-03-25T19:15:21.227 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.226+0000 7f8c27d13640 1 Processor -- start 2024-03-25T19:15:21.227 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.226+0000 7f8c27d13640 1 -- start start 2024-03-25T19:15:21.227 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.226+0000 7f8c27d13640 1 librados: setting wanted keys 2024-03-25T19:15:21.228 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.226+0000 7f8c27d13640 1 librados: calling monclient init 2024-03-25T19:15:21.228 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.226+0000 7f8c27d13640 10 monclient: init 2024-03-25T19:15:21.228 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.226+0000 7f8c27d13640 10 monclient: _reopen_session rank -1 2024-03-25T19:15:21.229 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.227+0000 7f8c27d13640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:15:21.229 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.227+0000 7f8c27d13640 10 monclient(hunting): picked mon.noname-a con 0x7f8c200b4670 addr v1:172.21.15.6:6789/0 2024-03-25T19:15:21.229 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.227+0000 7f8c27d13640 10 monclient(hunting): picked mon.noname-c con 0x7f8c200b4ff0 addr v1:172.21.15.6:6790/0 2024-03-25T19:15:21.229 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.227+0000 7f8c27d13640 10 monclient(hunting): picked mon.noname-b con 0x7f8c2013c480 addr v1:172.21.15.192:6789/0 2024-03-25T19:15:21.229 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.227+0000 7f8c27d13640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c200ac2b0 con 0x7f8c200b4670 2024-03-25T19:15:21.229 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.227+0000 7f8c27d13640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c2013be90 con 0x7f8c2013c480 2024-03-25T19:15:21.229 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.227+0000 7f8c27d13640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c2013c140 con 0x7f8c200b4ff0 2024-03-25T19:15:21.229 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.227+0000 7f8c27d13640 10 monclient(hunting): _renew_subs 2024-03-25T19:15:21.229 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.227+0000 7f8c27d13640 10 monclient(hunting): authenticate will time out at 6025.466856s 2024-03-25T19:15:21.241 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.241+0000 7f8c25aaf640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8c200b4670 0x7f8c2013b780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34954/0 (socket says 172.21.15.6:34954) 2024-03-25T19:15:21.241 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.241+0000 7f8c25aaf640 1 -- 172.21.15.6:0/1016595595 learned_addr learned my addr 172.21.15.6:0/1016595595 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:15:21.242 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.242+0000 7f8c167fc640 10 client.?.objecter ms_handle_connect 0x7f8c200b4670 2024-03-25T19:15:21.242 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.242+0000 7f8c167fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:15:21.245 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.245+0000 7f8c167fc640 10 client.?.objecter ms_handle_connect 0x7f8c2013c480 2024-03-25T19:15:21.245 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.245+0000 7f8c167fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:15:21.246 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.246+0000 7f8c167fc640 10 client.?.objecter ms_handle_connect 0x7f8c200b4ff0 2024-03-25T19:15:21.246 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.246+0000 7f8c167fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:15:21.252 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.253+0000 7f8c167fc640 1 -- 172.21.15.6:0/1016595595 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1182360944 0 0) 0x7f8c2013be90 con 0x7f8c2013c480 2024-03-25T19:15:21.252 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.253+0000 7f8c167fc640 10 monclient(hunting): _negotiate 2024-03-25T19:15:21.253 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.253+0000 7f8c167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:15:21.253 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.253+0000 7f8c167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:15:21.253 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.253+0000 7f8c167fc640 10 monclient(hunting): my global_id is 4279 2024-03-25T19:15:21.253 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.253+0000 7f8c167fc640 1 -- 172.21.15.6:0/1016595595 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c04004eb0 con 0x7f8c2013c480 2024-03-25T19:15:21.259 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.259+0000 7f8c167fc640 1 -- 172.21.15.6:0/1016595595 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3445732526 0 0) 0x7f8c04004eb0 con 0x7f8c2013c480 2024-03-25T19:15:21.259 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.259+0000 7f8c167fc640 1 -- 172.21.15.6:0/1016595595 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c04005300 con 0x7f8c2013c480 2024-03-25T19:15:21.260 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.259+0000 7f8c167fc640 1 -- 172.21.15.6:0/1016595595 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f8c18001e80 con 0x7f8c2013c480 2024-03-25T19:15:21.260 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.259+0000 7f8c167fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:15:21.260 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.259+0000 7f8c167fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:15:21.260 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.259+0000 7f8c167fc640 10 monclient(hunting): dump: 2024-03-25T19:15:21.260 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:15:21.260 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:21.260 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:21.260 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:21.260 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:15:21.260 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:21.260 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:15:21.260 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:15:21.260 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:15:21.260 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:21.260 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.259+0000 7f8c167fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:15:21.261 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.260+0000 7f8c167fc640 1 -- 172.21.15.6:0/1016595595 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4237778721 0 0) 0x7f8c2013c140 con 0x7f8c200b4ff0 2024-03-25T19:15:21.261 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.260+0000 7f8c167fc640 10 monclient(hunting): _negotiate 2024-03-25T19:15:21.261 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.260+0000 7f8c167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:15:21.261 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.260+0000 7f8c167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:15:21.261 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.260+0000 7f8c167fc640 10 monclient(hunting): my global_id is 4280 2024-03-25T19:15:21.261 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.260+0000 7f8c167fc640 1 -- 172.21.15.6:0/1016595595 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c04004eb0 con 0x7f8c200b4ff0 2024-03-25T19:15:21.263 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.263+0000 7f8c167fc640 1 -- 172.21.15.6:0/1016595595 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1921642984 0 0) 0x7f8c200ac2b0 con 0x7f8c200b4670 2024-03-25T19:15:21.263 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.263+0000 7f8c167fc640 10 monclient(hunting): _negotiate 2024-03-25T19:15:21.263 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.263+0000 7f8c167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:15:21.263 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.263+0000 7f8c167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:15:21.263 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.263+0000 7f8c167fc640 10 monclient(hunting): my global_id is 4275 2024-03-25T19:15:21.263 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.263+0000 7f8c167fc640 1 -- 172.21.15.6:0/1016595595 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c2013c140 con 0x7f8c200b4670 2024-03-25T19:15:21.282 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.282+0000 7f8c167fc640 1 -- 172.21.15.6:0/1016595595 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3311076855 0 0) 0x7f8c04005300 con 0x7f8c2013c480 2024-03-25T19:15:21.283 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.282+0000 7f8c167fc640 1 -- 172.21.15.6:0/1016595595 >> v1:172.21.15.6:6790/0 conn(0x7f8c200b4ff0 legacy=0x7f8c200ac920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:21.283 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.282+0000 7f8c167fc640 1 -- 172.21.15.6:0/1016595595 >> v1:172.21.15.6:6789/0 conn(0x7f8c200b4670 legacy=0x7f8c2013b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:21.283 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.282+0000 7f8c167fc640 10 monclient: _finish_hunting 0 2024-03-25T19:15:21.283 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.282+0000 7f8c167fc640 1 monclient: found mon.b 2024-03-25T19:15:21.283 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.282+0000 7f8c167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:15:21.283 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.282+0000 7f8c167fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:15:21.283 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.282+0000 7f8c167fc640 1 -- 172.21.15.6:0/1016595595 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c200ad030 con 0x7f8c2013c480 2024-03-25T19:15:21.283 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.282+0000 7f8c167fc640 10 monclient: _finish_auth 0 2024-03-25T19:15:21.283 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.282+0000 7f8c167fc640 10 monclient: _check_auth_tickets 2024-03-25T19:15:21.283 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.282+0000 7f8c167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:15:21.283 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.282+0000 7f8c27d13640 5 monclient: authenticate success, global_id 4279 2024-03-25T19:15:21.283 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.282+0000 7f8c27d13640 10 monclient: _renew_subs 2024-03-25T19:15:21.283 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.282+0000 7f8c27d13640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:15:21.283 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.282+0000 7f8c27d13640 1 -- 172.21.15.6:0/1016595595 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c2013a970 con 0x7f8c2013c480 2024-03-25T19:15:21.284 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.282+0000 7f8c27d13640 10 client.4279.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:15:21.284 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.282+0000 7f8c27d13640 10 monclient: _renew_subs 2024-03-25T19:15:21.284 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.282+0000 7f8c27d13640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:15:21.284 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.282+0000 7f8c27d13640 1 -- 172.21.15.6:0/1016595595 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c20139670 con 0x7f8c2013c480 2024-03-25T19:15:21.284 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.282+0000 7f8c27d13640 1 librados: init done 2024-03-25T19:15:21.284 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.283+0000 7f8c27d13640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:15:21.284 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.283+0000 7f8c27d13640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:15:21.284 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.283+0000 7f8c27d13640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:15:21.285 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.283+0000 7f8c27d13640 1 -- 172.21.15.6:0/1016595595 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c2006dbd0 con 0x7f8c2013c480 2024-03-25T19:15:21.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.551+0000 7f8c167fc640 1 -- 172.21.15.6:0/1016595595 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c18003c20 con 0x7f8c2013c480 2024-03-25T19:15:21.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.551+0000 7f8c167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:15:21.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.551+0000 7f8c167fc640 1 -- 172.21.15.6:0/1016595595 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f8c18003f30 con 0x7f8c2013c480 2024-03-25T19:15:21.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.551+0000 7f8c167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:15:21.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.551+0000 7f8c167fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:15:21.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.551+0000 7f8c167fc640 10 monclient: dump: 2024-03-25T19:15:21.552 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:15:21.552 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:21.552 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:21.552 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:21.552 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:15:21.552 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:21.552 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:15:21.552 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:15:21.553 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:15:21.553 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:21.553 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.551+0000 7f8c17fff640 4 set_mon_vals no callback set 2024-03-25T19:15:21.553 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.552+0000 7f8c167fc640 1 -- 172.21.15.6:0/1016595595 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f8c18019f00 con 0x7f8c2013c480 2024-03-25T19:15:21.553 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.553+0000 7f8c25aaf640 10 monclient: get_auth_request con 0x7f8c0406dfe0 auth_method 0 2024-03-25T19:15:21.553 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.553+0000 7f8c167fc640 10 client.4279.objecter ms_handle_connect 0x7f8c0406dfe0 2024-03-25T19:15:21.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.558+0000 7f8c167fc640 1 -- 172.21.15.6:0/1016595595 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f8c180538f0 con 0x7f8c2013c480 2024-03-25T19:15:21.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.558+0000 7f8c167fc640 10 client.4279.objecter ms_dispatch 0x7f8c20000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:15:21.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.558+0000 7f8c167fc640 3 client.4279.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:15:21.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.558+0000 7f8c167fc640 3 client.4279.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:15:21.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.558+0000 7f8c167fc640 20 client.4279.objecter dump_active .. 0 homeless 2024-03-25T19:15:21.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.559+0000 7f8c167fc640 1 -- 172.21.15.6:0/1016595595 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8c180b08b0 con 0x7f8c2013c480 2024-03-25T19:15:21.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.559+0000 7f8c167fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:15:21.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.559+0000 7f8c167fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:15:21.787 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.787+0000 7f8c27d13640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:15:21.787 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.787+0000 7f8c27d13640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:15:21.787 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.787+0000 7f8c27d13640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:15:21.787 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.787+0000 7f8c27d13640 1 -- 172.21.15.6:0/1016595595 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c2014f100 con 0x7f8c2013c480 2024-03-25T19:15:21.827 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.827+0000 7f8c167fc640 1 -- 172.21.15.6:0/1016595595 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f8c18052fc0 con 0x7f8c2013c480 2024-03-25T19:15:21.827 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.827+0000 7f8c167fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:15:21.827 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.827+0000 7f8c167fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:15:21.828 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:15:21.828 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:15:21.830 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.829+0000 7f8c27d13640 10 librados: watch_flush enter 2024-03-25T19:15:21.830 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.829+0000 7f8c27d13640 10 librados: watch_flush exit 2024-03-25T19:15:21.830 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.829+0000 7f8c27d13640 20 client.4279.objecter shutdown clearing up homeless session... 2024-03-25T19:15:21.830 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.829+0000 7f8c27d13640 10 client.4279.objecter successfully canceled tick 2024-03-25T19:15:21.830 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.830+0000 7f8c27d13640 1 -- 172.21.15.6:0/1016595595 >> v1:172.21.15.6:6800/2936652650 conn(0x7f8c0406dfe0 legacy=0x7f8c040704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:21.830 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.830+0000 7f8c27d13640 10 monclient: shutdown 2024-03-25T19:15:21.830 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.830+0000 7f8c27d13640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:15:21.830 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.830+0000 7f8c27d13640 1 -- 172.21.15.6:0/1016595595 >> v1:172.21.15.192:6789/0 conn(0x7f8c2013c480 legacy=0x7f8c201e2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:21.831 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.830+0000 7f8c27d13640 1 -- 172.21.15.6:0/1016595595 shutdown_connections 2024-03-25T19:15:21.831 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.830+0000 7f8c27d13640 1 -- 172.21.15.6:0/1016595595 >> 172.21.15.6:0/1016595595 conn(0x7f8c200617a0 msgr2=0x7f8c20145670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:15:21.831 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.830+0000 7f8c27d13640 1 -- 172.21.15.6:0/1016595595 shutdown_connections 2024-03-25T19:15:21.831 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.830+0000 7f8c27d13640 1 -- 172.21.15.6:0/1016595595 wait complete. 2024-03-25T19:15:21.831 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:21.830+0000 7f8c27d13640 1 librados: shutdown 2024-03-25T19:15:21.842 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:15:22.120 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:01:39.467 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:15:22.130 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:01:39.477 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:15:22.182 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:01:39.499 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:15:22.184 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:01:39.531 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:15:22.191 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:01:39.509 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:15:22.196 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:01:39.543 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:15:22.209 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:01:39.556 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:15:22.244 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:01:39.562 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:15:22.279 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:01:39.626 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:15:28.648 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:15:28.764 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.764+0000 7fc8b94a8640 10 monclient: get_monmap_and_config 2024-03-25T19:15:28.764 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.764+0000 7fc8b94a8640 10 monclient: build_initial_monmap 2024-03-25T19:15:28.764 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.764+0000 7fc8b94a8640 10 monclient: monmap: 2024-03-25T19:15:28.765 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:15:28.765 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:28.765 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:15:28.765624+0000 2024-03-25T19:15:28.765 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:15:28.765624+0000 2024-03-25T19:15:28.765 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:15:28.765 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:28.765 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:15:28.765 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:15:28.765 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:15:28.765 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:28.766 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.765+0000 7fc8b94a8640 1 Processor -- start 2024-03-25T19:15:28.766 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.765+0000 7fc8b94a8640 1 -- start start 2024-03-25T19:15:28.766 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.765+0000 7fc8b94a8640 10 monclient: init 2024-03-25T19:15:28.767 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.765+0000 7fc8b94a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:15:28.767 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.765+0000 7fc8b94a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:15:28.767 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.765+0000 7fc8b94a8640 10 monclient(hunting): picked mon.noname-a con 0x7fc8b40b4e70 addr v1:172.21.15.6:6789/0 2024-03-25T19:15:28.767 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.765+0000 7fc8b94a8640 10 monclient(hunting): picked mon.noname-b con 0x7fc8b40b44b0 addr v1:172.21.15.6:6790/0 2024-03-25T19:15:28.767 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.765+0000 7fc8b94a8640 10 monclient(hunting): picked mon.noname-c con 0x7fc8b40b1b00 addr v1:172.21.15.192:6789/0 2024-03-25T19:15:28.767 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.765+0000 7fc8b94a8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b40a96f0 con 0x7fc8b40b4e70 2024-03-25T19:15:28.767 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.765+0000 7fc8b94a8640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b40ab0e0 con 0x7fc8b40b1b00 2024-03-25T19:15:28.767 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.765+0000 7fc8b94a8640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b4145f70 con 0x7fc8b40b44b0 2024-03-25T19:15:28.767 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.765+0000 7fc8b94a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:15:28.767 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.765+0000 7fc8b94a8640 10 monclient(hunting): authenticate will time out at 6033.005734s 2024-03-25T19:15:28.787 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.787+0000 7fc8b37fe640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fc8b40b1b00 0x7fc8b414dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:33620/0 (socket says 172.21.15.6:33620) 2024-03-25T19:15:28.787 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.787+0000 7fc8b37fe640 1 -- 172.21.15.6:0/3638063493 learned_addr learned my addr 172.21.15.6:0/3638063493 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:15:28.802 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.802+0000 7fc8b1ffb640 1 -- 172.21.15.6:0/3638063493 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1345582578 0 0) 0x7fc8b40ab0e0 con 0x7fc8b40b1b00 2024-03-25T19:15:28.802 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.802+0000 7fc8b1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:15:28.802 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.802+0000 7fc8b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:15:28.802 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.802+0000 7fc8b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:15:28.802 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.802+0000 7fc8b1ffb640 10 monclient(hunting): my global_id is 4282 2024-03-25T19:15:28.802 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.802+0000 7fc8b1ffb640 1 -- 172.21.15.6:0/3638063493 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc89c003520 con 0x7fc8b40b1b00 2024-03-25T19:15:28.809 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.809+0000 7fc8b1ffb640 1 -- 172.21.15.6:0/3638063493 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4010593151 0 0) 0x7fc8b40a96f0 con 0x7fc8b40b4e70 2024-03-25T19:15:28.809 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.809+0000 7fc8b1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:15:28.809 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.809+0000 7fc8b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:15:28.809 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.809+0000 7fc8b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:15:28.809 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.809+0000 7fc8b1ffb640 10 monclient(hunting): my global_id is 4278 2024-03-25T19:15:28.810 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.809+0000 7fc8b1ffb640 1 -- 172.21.15.6:0/3638063493 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8b40ab0e0 con 0x7fc8b40b4e70 2024-03-25T19:15:28.813 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.813+0000 7fc8b1ffb640 1 -- 172.21.15.6:0/3638063493 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4237829930 0 0) 0x7fc89c003520 con 0x7fc8b40b1b00 2024-03-25T19:15:28.814 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.813+0000 7fc8b1ffb640 1 -- 172.21.15.6:0/3638063493 >> v1:172.21.15.6:6790/0 conn(0x7fc8b40b44b0 legacy=0x7fc8b4145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:28.814 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.813+0000 7fc8b1ffb640 1 -- 172.21.15.6:0/3638063493 >> v1:172.21.15.6:6789/0 conn(0x7fc8b40b4e70 legacy=0x7fc8b413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:28.814 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.813+0000 7fc8b1ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:15:28.814 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.813+0000 7fc8b1ffb640 1 monclient: found mon.noname-c 2024-03-25T19:15:28.814 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.813+0000 7fc8b1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:15:28.814 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.813+0000 7fc8b1ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:15:28.814 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.813+0000 7fc8b1ffb640 1 -- 172.21.15.6:0/3638063493 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b40a9b80 con 0x7fc8b40b1b00 2024-03-25T19:15:28.814 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.813+0000 7fc8b1ffb640 10 monclient: _finish_auth 0 2024-03-25T19:15:28.814 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.813+0000 7fc8b1ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:15:28.814 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.813+0000 7fc8b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:15:28.814 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.813+0000 7fc8b1ffb640 1 -- 172.21.15.6:0/3638063493 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fc8ac001a10 con 0x7fc8b40b1b00 2024-03-25T19:15:28.814 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.813+0000 7fc8b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:15:28.814 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.813+0000 7fc8b1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:15:28.815 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.813+0000 7fc8b1ffb640 10 monclient: dump: 2024-03-25T19:15:28.815 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:15:28.815 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:28.815 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:28.815 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:28.815 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:15:28.815 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:28.815 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:15:28.815 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:15:28.815 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:15:28.815 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:28.815 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.813+0000 7fc8b94a8640 5 monclient: authenticate success, global_id 4282 2024-03-25T19:15:28.815 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.813+0000 7fc8b94a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:15:28.837 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.837+0000 7fc8b1ffb640 1 -- 172.21.15.6:0/3638063493 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8ac001b70 con 0x7fc8b40b1b00 2024-03-25T19:15:28.837 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.837+0000 7fc8b1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:15:28.837 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.837+0000 7fc8b1ffb640 1 -- 172.21.15.6:0/3638063493 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fc8ac001f80 con 0x7fc8b40b1b00 2024-03-25T19:15:28.837 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.837+0000 7fc8b94a8640 10 monclient: get_monmap_and_config success 2024-03-25T19:15:28.837 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.837+0000 7fc8b94a8640 4 set_mon_vals no callback set 2024-03-25T19:15:28.837 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.837+0000 7fc8b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:15:28.838 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.837+0000 7fc8b1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:15:28.838 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.837+0000 7fc8b1ffb640 10 monclient: dump: 2024-03-25T19:15:28.838 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:15:28.838 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:28.838 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:28.838 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:28.838 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:15:28.838 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:28.838 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:15:28.838 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:15:28.838 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:15:28.838 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:28.840 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.838+0000 7fc8b94a8640 10 monclient: shutdown 2024-03-25T19:15:28.840 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.838+0000 7fc8b94a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:15:28.840 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.838+0000 7fc8b94a8640 1 -- 172.21.15.6:0/3638063493 >> v1:172.21.15.192:6789/0 conn(0x7fc8b40b1b00 legacy=0x7fc8b414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:28.840 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.838+0000 7fc8b94a8640 1 -- 172.21.15.6:0/3638063493 shutdown_connections 2024-03-25T19:15:28.840 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.838+0000 7fc8b94a8640 1 -- 172.21.15.6:0/3638063493 >> 172.21.15.6:0/3638063493 conn(0x7fc8b40617a0 msgr2=0x7fc8b4061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:15:28.840 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.838+0000 7fc8b94a8640 1 -- 172.21.15.6:0/3638063493 shutdown_connections 2024-03-25T19:15:28.840 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.838+0000 7fc8b94a8640 1 -- 172.21.15.6:0/3638063493 wait complete. 2024-03-25T19:15:28.840 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.839+0000 7fc8b94a8640 10 monclient: build_initial_monmap 2024-03-25T19:15:28.840 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.839+0000 7fc8b94a8640 10 monclient: monmap: 2024-03-25T19:15:28.840 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:15:28.840 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:15:28.841 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:15:28.841 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:15:28.841 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:15:28.841 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:15:28.841 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:15:28.841 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:15:28.841 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:15:28.841 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:28.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.839+0000 7fc8b94a8640 1 librados: starting msgr at 2024-03-25T19:15:28.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.839+0000 7fc8b94a8640 1 librados: starting objecter 2024-03-25T19:15:28.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.839+0000 7fc8b94a8640 1 Processor -- start 2024-03-25T19:15:28.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.839+0000 7fc8b94a8640 1 -- start start 2024-03-25T19:15:28.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.839+0000 7fc8b94a8640 1 librados: setting wanted keys 2024-03-25T19:15:28.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.839+0000 7fc8b94a8640 1 librados: calling monclient init 2024-03-25T19:15:28.842 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.839+0000 7fc8b94a8640 10 monclient: init 2024-03-25T19:15:28.842 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.840+0000 7fc8b94a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:15:28.842 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.840+0000 7fc8b94a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:15:28.842 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.840+0000 7fc8b94a8640 10 monclient(hunting): picked mon.noname-c con 0x7fc8b40b1b00 addr v1:172.21.15.6:6790/0 2024-03-25T19:15:28.843 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.840+0000 7fc8b94a8640 10 monclient(hunting): picked mon.noname-a con 0x7fc8b40b44b0 addr v1:172.21.15.6:6789/0 2024-03-25T19:15:28.843 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.840+0000 7fc8b94a8640 10 monclient(hunting): picked mon.noname-b con 0x7fc8b40b4e70 addr v1:172.21.15.192:6789/0 2024-03-25T19:15:28.843 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.840+0000 7fc8b94a8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b41d77f0 con 0x7fc8b40b44b0 2024-03-25T19:15:28.843 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.840+0000 7fc8b94a8640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b4136e60 con 0x7fc8b40b4e70 2024-03-25T19:15:28.843 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.840+0000 7fc8b94a8640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b41d8670 con 0x7fc8b40b1b00 2024-03-25T19:15:28.843 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.840+0000 7fc8b94a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:15:28.843 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.840+0000 7fc8b94a8640 10 monclient(hunting): authenticate will time out at 6033.080653s 2024-03-25T19:15:28.852 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.852+0000 7fc8b2ffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fc8b40b1b00 0x7fc8b4136750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:40650/0 (socket says 172.21.15.6:40650) 2024-03-25T19:15:28.852 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.852+0000 7fc8b2ffd640 1 -- 172.21.15.6:0/3804979649 learned_addr learned my addr 172.21.15.6:0/3804979649 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:15:28.853 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.853+0000 7fc89b7fe640 10 client.?.objecter ms_handle_connect 0x7fc8b40b1b00 2024-03-25T19:15:28.853 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.853+0000 7fc89b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:15:28.853 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.853+0000 7fc89b7fe640 10 client.?.objecter ms_handle_connect 0x7fc8b40b4e70 2024-03-25T19:15:28.853 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.853+0000 7fc89b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:15:28.868 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.868+0000 7fc89b7fe640 10 client.?.objecter ms_handle_connect 0x7fc8b40b44b0 2024-03-25T19:15:28.868 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.868+0000 7fc89b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:15:28.868 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.868+0000 7fc89b7fe640 1 -- 172.21.15.6:0/3804979649 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1745073787 0 0) 0x7fc8b41d8670 con 0x7fc8b40b1b00 2024-03-25T19:15:28.868 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.868+0000 7fc89b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:15:28.868 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.868+0000 7fc89b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:15:28.868 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.868+0000 7fc89b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:15:28.868 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.868+0000 7fc89b7fe640 10 monclient(hunting): my global_id is 4286 2024-03-25T19:15:28.868 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.868+0000 7fc89b7fe640 1 -- 172.21.15.6:0/3804979649 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8a0004eb0 con 0x7fc8b40b1b00 2024-03-25T19:15:28.869 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.869+0000 7fc89b7fe640 1 -- 172.21.15.6:0/3804979649 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2054125126 0 0) 0x7fc8b4136e60 con 0x7fc8b40b4e70 2024-03-25T19:15:28.869 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.869+0000 7fc89b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:15:28.869 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.869+0000 7fc89b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:15:28.869 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.869+0000 7fc89b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:15:28.869 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.869+0000 7fc89b7fe640 10 monclient(hunting): my global_id is 4285 2024-03-25T19:15:28.869 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.869+0000 7fc89b7fe640 1 -- 172.21.15.6:0/3804979649 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8b41d8670 con 0x7fc8b40b4e70 2024-03-25T19:15:28.896 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.896+0000 7fc89b7fe640 1 -- 172.21.15.6:0/3804979649 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4113828436 0 0) 0x7fc8b41d77f0 con 0x7fc8b40b44b0 2024-03-25T19:15:28.896 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.896+0000 7fc89b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:15:28.896 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.896+0000 7fc89b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:15:28.896 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.896+0000 7fc89b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:15:28.896 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.896+0000 7fc89b7fe640 10 monclient(hunting): my global_id is 4281 2024-03-25T19:15:28.896 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.896+0000 7fc89b7fe640 1 -- 172.21.15.6:0/3804979649 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8b4136e60 con 0x7fc8b40b44b0 2024-03-25T19:15:28.962 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.962+0000 7fc89b7fe640 1 -- 172.21.15.6:0/3804979649 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2786094251 0 0) 0x7fc8b4136e60 con 0x7fc8b40b44b0 2024-03-25T19:15:28.962 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.962+0000 7fc89b7fe640 1 -- 172.21.15.6:0/3804979649 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8b41d77f0 con 0x7fc8b40b44b0 2024-03-25T19:15:28.962 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.962+0000 7fc89b7fe640 1 -- 172.21.15.6:0/3804979649 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fc8a8001830 con 0x7fc8b40b44b0 2024-03-25T19:15:28.963 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.962+0000 7fc89b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:15:28.963 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.962+0000 7fc89b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:15:28.963 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.962+0000 7fc89b7fe640 10 monclient(hunting): dump: 2024-03-25T19:15:28.963 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:15:28.963 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:28.963 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:28.963 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:28.963 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:15:28.963 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:28.963 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:15:28.963 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:15:28.963 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:15:28.963 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:28.963 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.962+0000 7fc89b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:15:28.966 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.966+0000 7fc89b7fe640 1 -- 172.21.15.6:0/3804979649 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2624640434 0 0) 0x7fc8b41d8670 con 0x7fc8b40b4e70 2024-03-25T19:15:28.966 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.966+0000 7fc89b7fe640 1 -- 172.21.15.6:0/3804979649 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8b4136e60 con 0x7fc8b40b4e70 2024-03-25T19:15:28.966 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.966+0000 7fc89b7fe640 1 -- 172.21.15.6:0/3804979649 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fc8ac001e80 con 0x7fc8b40b4e70 2024-03-25T19:15:28.966 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.966+0000 7fc89b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:15:28.966 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.966+0000 7fc89b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:15:28.966 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.966+0000 7fc89b7fe640 10 monclient(hunting): dump: 2024-03-25T19:15:28.966 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:15:28.966 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:28.966 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:28.967 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:28.967 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:15:28.967 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:28.967 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:15:28.967 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:15:28.967 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:15:28.967 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:28.987 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.987+0000 7fc89b7fe640 1 -- 172.21.15.6:0/3804979649 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 821893365 0 0) 0x7fc8b4136e60 con 0x7fc8b40b4e70 2024-03-25T19:15:28.987 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.987+0000 7fc89b7fe640 1 -- 172.21.15.6:0/3804979649 >> v1:172.21.15.6:6790/0 conn(0x7fc8b40b1b00 legacy=0x7fc8b4136750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:28.987 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.987+0000 7fc89b7fe640 1 -- 172.21.15.6:0/3804979649 >> v1:172.21.15.6:6789/0 conn(0x7fc8b40b44b0 legacy=0x7fc8b41d7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:28.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.987+0000 7fc89b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:15:28.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.987+0000 7fc89b7fe640 1 monclient: found mon.b 2024-03-25T19:15:28.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.987+0000 7fc89b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:15:28.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.987+0000 7fc89b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:15:28.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.987+0000 7fc89b7fe640 1 -- 172.21.15.6:0/3804979649 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b41380a0 con 0x7fc8b40b4e70 2024-03-25T19:15:28.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.987+0000 7fc89b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:15:28.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.987+0000 7fc89b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:15:28.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.987+0000 7fc89b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:15:28.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.987+0000 7fc8b94a8640 5 monclient: authenticate success, global_id 4285 2024-03-25T19:15:28.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.987+0000 7fc8b94a8640 10 monclient: _renew_subs 2024-03-25T19:15:28.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.987+0000 7fc8b94a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:15:28.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.987+0000 7fc8b94a8640 1 -- 172.21.15.6:0/3804979649 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8b41d7a40 con 0x7fc8b40b4e70 2024-03-25T19:15:28.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.987+0000 7fc8b94a8640 10 client.4285.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:15:28.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.987+0000 7fc8b94a8640 10 monclient: _renew_subs 2024-03-25T19:15:28.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.987+0000 7fc8b94a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:15:28.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.987+0000 7fc8b94a8640 1 -- 172.21.15.6:0/3804979649 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8b4134690 con 0x7fc8b40b4e70 2024-03-25T19:15:28.989 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.987+0000 7fc8b94a8640 1 librados: init done 2024-03-25T19:15:28.989 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.988+0000 7fc8b94a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:15:28.989 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.988+0000 7fc8b94a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:15:28.989 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.988+0000 7fc8b94a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:15:28.989 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:28.988+0000 7fc8b94a8640 1 -- 172.21.15.6:0/3804979649 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8b404f730 con 0x7fc8b40b4e70 2024-03-25T19:15:29.258 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:29.254+0000 7fc89b7fe640 1 -- 172.21.15.6:0/3804979649 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8ac0015c0 con 0x7fc8b40b4e70 2024-03-25T19:15:29.258 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:29.254+0000 7fc89b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:15:29.258 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:29.254+0000 7fc89b7fe640 1 -- 172.21.15.6:0/3804979649 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fc8ac004fb0 con 0x7fc8b40b4e70 2024-03-25T19:15:29.258 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:29.254+0000 7fc89b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:15:29.258 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:29.254+0000 7fc89b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:15:29.258 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:29.254+0000 7fc89b7fe640 10 monclient: dump: 2024-03-25T19:15:29.258 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:15:29.259 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:29.259 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:29.259 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:29.259 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:15:29.259 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:29.259 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:15:29.259 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:15:29.259 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:15:29.259 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:29.259 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:29.254+0000 7fc8b17fa640 4 set_mon_vals no callback set 2024-03-25T19:15:29.259 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:29.255+0000 7fc89b7fe640 1 -- 172.21.15.6:0/3804979649 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7fc8ac01af80 con 0x7fc8b40b4e70 2024-03-25T19:15:29.259 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:29.256+0000 7fc8b2ffd640 10 monclient: get_auth_request con 0x7fc8a006df80 auth_method 0 2024-03-25T19:15:29.259 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:29.256+0000 7fc89b7fe640 10 client.4285.objecter ms_handle_connect 0x7fc8a006df80 2024-03-25T19:15:29.261 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:29.261+0000 7fc89b7fe640 1 -- 172.21.15.6:0/3804979649 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7fc8ac054a70 con 0x7fc8b40b4e70 2024-03-25T19:15:29.262 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:29.261+0000 7fc89b7fe640 10 client.4285.objecter ms_dispatch 0x7fc8b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:15:29.262 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:29.261+0000 7fc89b7fe640 3 client.4285.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:15:29.262 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:29.261+0000 7fc89b7fe640 3 client.4285.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:15:29.262 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:29.262+0000 7fc89b7fe640 20 client.4285.objecter dump_active .. 0 homeless 2024-03-25T19:15:29.262 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:29.262+0000 7fc89b7fe640 1 -- 172.21.15.6:0/3804979649 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc8ac0503b0 con 0x7fc8b40b4e70 2024-03-25T19:15:29.262 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:29.262+0000 7fc89b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:15:29.262 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:29.262+0000 7fc89b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:15:29.508 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:29.508+0000 7fc8b94a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:15:29.509 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:29.508+0000 7fc8b94a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:15:29.509 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:29.508+0000 7fc8b94a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:15:29.509 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:29.508+0000 7fc8b94a8640 1 -- 172.21.15.6:0/3804979649 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8b41d7bd0 con 0x7fc8b40b4e70 2024-03-25T19:15:29.532 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:29.532+0000 7fc89b7fe640 1 -- 172.21.15.6:0/3804979649 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7fc8ac0524b0 con 0x7fc8b40b4e70 2024-03-25T19:15:29.532 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:29.532+0000 7fc89b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:15:29.532 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:29.532+0000 7fc89b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:15:29.532 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:15:29.532 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:15:29.535 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:29.535+0000 7fc8b94a8640 10 librados: watch_flush enter 2024-03-25T19:15:29.535 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:29.535+0000 7fc8b94a8640 10 librados: watch_flush exit 2024-03-25T19:15:29.536 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:29.535+0000 7fc8b94a8640 20 client.4285.objecter shutdown clearing up homeless session... 2024-03-25T19:15:29.536 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:29.535+0000 7fc8b94a8640 10 client.4285.objecter successfully canceled tick 2024-03-25T19:15:29.536 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:29.535+0000 7fc8b94a8640 1 -- 172.21.15.6:0/3804979649 >> v1:172.21.15.6:6800/2936652650 conn(0x7fc8a006df80 legacy=0x7fc8a0070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:29.536 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:29.535+0000 7fc8b94a8640 10 monclient: shutdown 2024-03-25T19:15:29.536 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:29.535+0000 7fc8b94a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:15:29.536 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:29.535+0000 7fc8b94a8640 1 -- 172.21.15.6:0/3804979649 >> v1:172.21.15.192:6789/0 conn(0x7fc8b40b4e70 legacy=0x7fc8b4133f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:29.536 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:29.535+0000 7fc8b94a8640 1 -- 172.21.15.6:0/3804979649 shutdown_connections 2024-03-25T19:15:29.536 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:29.535+0000 7fc8b94a8640 1 -- 172.21.15.6:0/3804979649 >> 172.21.15.6:0/3804979649 conn(0x7fc8b40617a0 msgr2=0x7fc8b40b3100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:15:29.536 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:29.535+0000 7fc8b94a8640 1 -- 172.21.15.6:0/3804979649 shutdown_connections 2024-03-25T19:15:29.536 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:29.535+0000 7fc8b94a8640 1 -- 172.21.15.6:0/3804979649 wait complete. 2024-03-25T19:15:29.536 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:29.535+0000 7fc8b94a8640 1 librados: shutdown 2024-03-25T19:15:29.547 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:15:36.354 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:15:36.469 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.469+0000 7f1f05e9a640 10 monclient: get_monmap_and_config 2024-03-25T19:15:36.469 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.469+0000 7f1f05e9a640 10 monclient: build_initial_monmap 2024-03-25T19:15:36.469 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.469+0000 7f1f05e9a640 10 monclient: monmap: 2024-03-25T19:15:36.469 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:15:36.469 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:36.469 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:15:36.470302+0000 2024-03-25T19:15:36.469 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:15:36.470302+0000 2024-03-25T19:15:36.469 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:15:36.469 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:36.469 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:15:36.470 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:15:36.470 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:15:36.470 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:36.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.470+0000 7f1f05e9a640 1 Processor -- start 2024-03-25T19:15:36.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.470+0000 7f1f05e9a640 1 -- start start 2024-03-25T19:15:36.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.470+0000 7f1f05e9a640 10 monclient: init 2024-03-25T19:15:36.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.470+0000 7f1f05e9a640 10 monclient: _reopen_session rank -1 2024-03-25T19:15:36.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.470+0000 7f1f05e9a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:15:36.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.470+0000 7f1f05e9a640 10 monclient(hunting): picked mon.noname-b con 0x7f1f000b4ff0 addr v1:172.21.15.6:6790/0 2024-03-25T19:15:36.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.470+0000 7f1f05e9a640 10 monclient(hunting): picked mon.noname-a con 0x7f1f000b4670 addr v1:172.21.15.6:6789/0 2024-03-25T19:15:36.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.470+0000 7f1f05e9a640 10 monclient(hunting): picked mon.noname-c con 0x7f1f0013c480 addr v1:172.21.15.192:6789/0 2024-03-25T19:15:36.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.470+0000 7f1f05e9a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f000ab0e0 con 0x7f1f000b4670 2024-03-25T19:15:36.472 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.470+0000 7f1f05e9a640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f000ab3c0 con 0x7f1f0013c480 2024-03-25T19:15:36.472 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.470+0000 7f1f05e9a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f000a9f30 con 0x7f1f000b4ff0 2024-03-25T19:15:36.472 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.470+0000 7f1f05e9a640 10 monclient(hunting): _renew_subs 2024-03-25T19:15:36.472 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.470+0000 7f1f05e9a640 10 monclient(hunting): authenticate will time out at 6040.710693s 2024-03-25T19:15:36.472 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.472+0000 7f1efffff640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f1f0013c480 0x7f1f0014dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:38312/0 (socket says 172.21.15.6:38312) 2024-03-25T19:15:36.472 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.472+0000 7f1efffff640 1 -- 172.21.15.6:0/363875669 learned_addr learned my addr 172.21.15.6:0/363875669 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:15:36.516 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.516+0000 7f1efe7fc640 1 -- 172.21.15.6:0/363875669 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3634316789 0 0) 0x7f1f000ab0e0 con 0x7f1f000b4670 2024-03-25T19:15:36.516 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.516+0000 7f1efe7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:15:36.516 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.516+0000 7f1efe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:15:36.517 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.516+0000 7f1efe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:15:36.517 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.516+0000 7f1efe7fc640 10 monclient(hunting): my global_id is 4284 2024-03-25T19:15:36.517 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.516+0000 7f1efe7fc640 1 -- 172.21.15.6:0/363875669 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ee8003520 con 0x7f1f000b4670 2024-03-25T19:15:36.528 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.528+0000 7f1efe7fc640 1 -- 172.21.15.6:0/363875669 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3048812577 0 0) 0x7f1f000a9f30 con 0x7f1f000b4ff0 2024-03-25T19:15:36.528 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.528+0000 7f1efe7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:15:36.528 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.528+0000 7f1efe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:15:36.528 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.528+0000 7f1efe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:15:36.528 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.528+0000 7f1efe7fc640 10 monclient(hunting): my global_id is 4289 2024-03-25T19:15:36.528 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.528+0000 7f1efe7fc640 1 -- 172.21.15.6:0/363875669 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f000ab0e0 con 0x7f1f000b4ff0 2024-03-25T19:15:36.544 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.544+0000 7f1efe7fc640 1 -- 172.21.15.6:0/363875669 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4185001333 0 0) 0x7f1ee8003520 con 0x7f1f000b4670 2024-03-25T19:15:36.544 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.544+0000 7f1efe7fc640 1 -- 172.21.15.6:0/363875669 >> v1:172.21.15.6:6790/0 conn(0x7f1f000b4ff0 legacy=0x7f1f0013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:36.544 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.544+0000 7f1efe7fc640 1 -- 172.21.15.6:0/363875669 >> v1:172.21.15.192:6789/0 conn(0x7f1f0013c480 legacy=0x7f1f0014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:36.544 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.544+0000 7f1efe7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:15:36.544 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.544+0000 7f1efe7fc640 1 monclient: found mon.noname-a 2024-03-25T19:15:36.544 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.544+0000 7f1efe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:15:36.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.544+0000 7f1efe7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-03-25T19:15:36.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.544+0000 7f1efe7fc640 1 -- 172.21.15.6:0/363875669 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f000a9b70 con 0x7f1f000b4670 2024-03-25T19:15:36.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.544+0000 7f1efe7fc640 10 monclient: _finish_auth 0 2024-03-25T19:15:36.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.544+0000 7f1efe7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:15:36.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.544+0000 7f1efe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:15:36.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.544+0000 7f1efe7fc640 1 -- 172.21.15.6:0/363875669 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f1ef4001690 con 0x7f1f000b4670 2024-03-25T19:15:36.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.544+0000 7f1efe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:15:36.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.544+0000 7f1efe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:15:36.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.544+0000 7f1efe7fc640 10 monclient: dump: 2024-03-25T19:15:36.545 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:15:36.545 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:36.545 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:36.545 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:36.545 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:15:36.546 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:36.546 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:15:36.546 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:15:36.546 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:15:36.546 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:36.546 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.544+0000 7f1f05e9a640 5 monclient: authenticate success, global_id 4284 2024-03-25T19:15:36.546 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.544+0000 7f1f05e9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:15:36.569 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.569+0000 7f1efe7fc640 1 -- 172.21.15.6:0/363875669 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ef40017f0 con 0x7f1f000b4670 2024-03-25T19:15:36.570 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.569+0000 7f1efe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:15:36.570 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.569+0000 7f1efe7fc640 1 -- 172.21.15.6:0/363875669 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f1ef4001c00 con 0x7f1f000b4670 2024-03-25T19:15:36.570 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.569+0000 7f1efe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:15:36.570 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.569+0000 7f1efe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:15:36.570 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.569+0000 7f1efe7fc640 10 monclient: dump: 2024-03-25T19:15:36.570 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:15:36.570 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:36.570 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:36.570 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:36.570 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:15:36.570 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:36.570 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:15:36.570 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:15:36.570 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:15:36.571 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:36.571 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.570+0000 7f1f05e9a640 10 monclient: get_monmap_and_config success 2024-03-25T19:15:36.571 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.570+0000 7f1f05e9a640 4 set_mon_vals no callback set 2024-03-25T19:15:36.572 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.570+0000 7f1f05e9a640 10 monclient: shutdown 2024-03-25T19:15:36.572 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.570+0000 7f1f05e9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:15:36.572 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.570+0000 7f1f05e9a640 1 -- 172.21.15.6:0/363875669 >> v1:172.21.15.6:6789/0 conn(0x7f1f000b4670 legacy=0x7f1f001425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:36.573 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.571+0000 7f1f05e9a640 1 -- 172.21.15.6:0/363875669 shutdown_connections 2024-03-25T19:15:36.573 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.571+0000 7f1f05e9a640 1 -- 172.21.15.6:0/363875669 >> 172.21.15.6:0/363875669 conn(0x7f1f000617a0 msgr2=0x7f1f00061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:15:36.573 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.571+0000 7f1f05e9a640 1 -- 172.21.15.6:0/363875669 shutdown_connections 2024-03-25T19:15:36.573 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.571+0000 7f1f05e9a640 1 -- 172.21.15.6:0/363875669 wait complete. 2024-03-25T19:15:36.573 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.571+0000 7f1f05e9a640 10 monclient: build_initial_monmap 2024-03-25T19:15:36.573 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.571+0000 7f1f05e9a640 10 monclient: monmap: 2024-03-25T19:15:36.573 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:15:36.573 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:15:36.573 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:15:36.573 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:15:36.573 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:15:36.573 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:15:36.573 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:15:36.573 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:15:36.573 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:15:36.573 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:36.574 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.572+0000 7f1f05e9a640 1 librados: starting msgr at 2024-03-25T19:15:36.574 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.572+0000 7f1f05e9a640 1 librados: starting objecter 2024-03-25T19:15:36.574 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.572+0000 7f1f05e9a640 1 Processor -- start 2024-03-25T19:15:36.574 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.572+0000 7f1f05e9a640 1 -- start start 2024-03-25T19:15:36.574 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.572+0000 7f1f05e9a640 1 librados: setting wanted keys 2024-03-25T19:15:36.574 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.572+0000 7f1f05e9a640 1 librados: calling monclient init 2024-03-25T19:15:36.574 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.572+0000 7f1f05e9a640 10 monclient: init 2024-03-25T19:15:36.575 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.573+0000 7f1f05e9a640 10 monclient: _reopen_session rank -1 2024-03-25T19:15:36.575 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.573+0000 7f1f05e9a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:15:36.575 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.573+0000 7f1f05e9a640 10 monclient(hunting): picked mon.noname-a con 0x7f1f000b4670 addr v1:172.21.15.6:6789/0 2024-03-25T19:15:36.575 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.573+0000 7f1f05e9a640 10 monclient(hunting): picked mon.noname-b con 0x7f1f000b4ff0 addr v1:172.21.15.192:6789/0 2024-03-25T19:15:36.575 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.574+0000 7f1f05e9a640 10 monclient(hunting): picked mon.noname-c con 0x7f1f0013c480 addr v1:172.21.15.6:6790/0 2024-03-25T19:15:36.575 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.574+0000 7f1f05e9a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f00144740 con 0x7f1f000b4670 2024-03-25T19:15:36.576 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.574+0000 7f1f05e9a640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f00130b60 con 0x7f1f000b4ff0 2024-03-25T19:15:36.576 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.574+0000 7f1f05e9a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f0013c930 con 0x7f1f0013c480 2024-03-25T19:15:36.576 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.574+0000 7f1f05e9a640 10 monclient(hunting): _renew_subs 2024-03-25T19:15:36.576 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.574+0000 7f1f05e9a640 10 monclient(hunting): authenticate will time out at 6040.814075s 2024-03-25T19:15:36.576 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.577+0000 7f1efeffd640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f1f000b4ff0 0x7f1f0012fd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:38390/0 (socket says 172.21.15.6:38390) 2024-03-25T19:15:36.577 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.577+0000 7f1efeffd640 1 -- 172.21.15.6:0/2771991392 learned_addr learned my addr 172.21.15.6:0/2771991392 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:15:36.577 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.578+0000 7f1ee7fff640 10 client.?.objecter ms_handle_connect 0x7f1f000b4ff0 2024-03-25T19:15:36.577 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.578+0000 7f1ee7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:15:36.579 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.579+0000 7f1ee7fff640 10 client.?.objecter ms_handle_connect 0x7f1f0013c480 2024-03-25T19:15:36.579 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.579+0000 7f1ee7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:15:36.654 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.654+0000 7f1ee7fff640 10 client.?.objecter ms_handle_connect 0x7f1f000b4670 2024-03-25T19:15:36.654 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.654+0000 7f1ee7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:15:36.666 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.666+0000 7f1ee7fff640 1 -- 172.21.15.6:0/2771991392 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 262593008 0 0) 0x7f1f0013c930 con 0x7f1f0013c480 2024-03-25T19:15:36.666 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.666+0000 7f1ee7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:15:36.666 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.666+0000 7f1ee7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:15:36.666 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.666+0000 7f1ee7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:15:36.666 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.666+0000 7f1ee7fff640 10 monclient(hunting): my global_id is 4292 2024-03-25T19:15:36.666 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.666+0000 7f1ee7fff640 1 -- 172.21.15.6:0/2771991392 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1eec004eb0 con 0x7f1f0013c480 2024-03-25T19:15:36.669 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.669+0000 7f1ee7fff640 1 -- 172.21.15.6:0/2771991392 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1523213623 0 0) 0x7f1f00144740 con 0x7f1f000b4670 2024-03-25T19:15:36.669 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.669+0000 7f1ee7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:15:36.669 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.669+0000 7f1ee7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:15:36.669 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.669+0000 7f1ee7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:15:36.669 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.669+0000 7f1ee7fff640 10 monclient(hunting): my global_id is 4287 2024-03-25T19:15:36.669 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.669+0000 7f1ee7fff640 1 -- 172.21.15.6:0/2771991392 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f0013c930 con 0x7f1f000b4670 2024-03-25T19:15:36.675 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.675+0000 7f1ee7fff640 1 -- 172.21.15.6:0/2771991392 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 334098940 0 0) 0x7f1f00130b60 con 0x7f1f000b4ff0 2024-03-25T19:15:36.675 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.675+0000 7f1ee7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:15:36.675 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.675+0000 7f1ee7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:15:36.675 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.675+0000 7f1ee7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:15:36.676 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.675+0000 7f1ee7fff640 10 monclient(hunting): my global_id is 4288 2024-03-25T19:15:36.676 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.675+0000 7f1ee7fff640 1 -- 172.21.15.6:0/2771991392 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f00144740 con 0x7f1f000b4ff0 2024-03-25T19:15:36.698 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.698+0000 7f1ee7fff640 1 -- 172.21.15.6:0/2771991392 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2042838656 0 0) 0x7f1eec004eb0 con 0x7f1f0013c480 2024-03-25T19:15:36.698 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.698+0000 7f1ee7fff640 1 -- 172.21.15.6:0/2771991392 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f00130b60 con 0x7f1f0013c480 2024-03-25T19:15:36.698 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.698+0000 7f1ee7fff640 1 -- 172.21.15.6:0/2771991392 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f1ef8001b00 con 0x7f1f0013c480 2024-03-25T19:15:36.699 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.698+0000 7f1ee7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:15:36.699 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.698+0000 7f1ee7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:15:36.699 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.698+0000 7f1ee7fff640 10 monclient(hunting): dump: 2024-03-25T19:15:36.699 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:15:36.699 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:36.699 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:36.699 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:36.699 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:15:36.699 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:36.699 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:15:36.699 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:15:36.699 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:15:36.699 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:36.699 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.698+0000 7f1ee7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:15:36.700 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.699+0000 7f1ee7fff640 1 -- 172.21.15.6:0/2771991392 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2655523191 0 0) 0x7f1f0013c930 con 0x7f1f000b4670 2024-03-25T19:15:36.700 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.699+0000 7f1ee7fff640 1 -- 172.21.15.6:0/2771991392 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1eec004eb0 con 0x7f1f000b4670 2024-03-25T19:15:36.700 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.699+0000 7f1ee7fff640 1 -- 172.21.15.6:0/2771991392 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f1ef0001930 con 0x7f1f000b4670 2024-03-25T19:15:36.700 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.699+0000 7f1ee7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:15:36.700 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.699+0000 7f1ee7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:15:36.700 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.699+0000 7f1ee7fff640 10 monclient(hunting): dump: 2024-03-25T19:15:36.700 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:15:36.700 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:36.701 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:36.701 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:36.701 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:15:36.701 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:36.701 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:15:36.701 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:15:36.701 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:15:36.701 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:36.703 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.703+0000 7f1ee7fff640 1 -- 172.21.15.6:0/2771991392 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 285893802 0 0) 0x7f1f00144740 con 0x7f1f000b4ff0 2024-03-25T19:15:36.703 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.703+0000 7f1ee7fff640 1 -- 172.21.15.6:0/2771991392 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f0013c930 con 0x7f1f000b4ff0 2024-03-25T19:15:36.703 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.703+0000 7f1ee7fff640 1 -- 172.21.15.6:0/2771991392 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f1ef4001650 con 0x7f1f000b4ff0 2024-03-25T19:15:36.703 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.703+0000 7f1ee7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:15:36.703 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.703+0000 7f1ee7fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:15:36.703 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.703+0000 7f1ee7fff640 10 monclient(hunting): dump: 2024-03-25T19:15:36.703 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:15:36.703 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:36.703 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:36.703 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:36.703 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:15:36.703 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:36.703 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:15:36.703 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:15:36.704 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:15:36.704 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:36.720 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.720+0000 7f1ee7fff640 1 -- 172.21.15.6:0/2771991392 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 331885853 0 0) 0x7f1f0013c930 con 0x7f1f000b4ff0 2024-03-25T19:15:36.720 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.720+0000 7f1ee7fff640 1 -- 172.21.15.6:0/2771991392 >> v1:172.21.15.6:6790/0 conn(0x7f1f0013c480 legacy=0x7f1f00130450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:36.720 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.720+0000 7f1ee7fff640 1 -- 172.21.15.6:0/2771991392 >> v1:172.21.15.6:6789/0 conn(0x7f1f000b4670 legacy=0x7f1f00145d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:36.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.720+0000 7f1ee7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:15:36.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.720+0000 7f1ee7fff640 1 monclient: found mon.b 2024-03-25T19:15:36.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.720+0000 7f1ee7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:15:36.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.720+0000 7f1ee7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:15:36.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.720+0000 7f1ee7fff640 1 -- 172.21.15.6:0/2771991392 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f00144970 con 0x7f1f000b4ff0 2024-03-25T19:15:36.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.720+0000 7f1ee7fff640 10 monclient: _finish_auth 0 2024-03-25T19:15:36.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.720+0000 7f1ee7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:15:36.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.720+0000 7f1ee7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:15:36.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.720+0000 7f1f05e9a640 5 monclient: authenticate success, global_id 4288 2024-03-25T19:15:36.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.720+0000 7f1f05e9a640 10 monclient: _renew_subs 2024-03-25T19:15:36.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.720+0000 7f1f05e9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:15:36.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.720+0000 7f1f05e9a640 1 -- 172.21.15.6:0/2771991392 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f00144c50 con 0x7f1f000b4ff0 2024-03-25T19:15:36.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.720+0000 7f1f05e9a640 10 client.4288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:15:36.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.720+0000 7f1f05e9a640 10 monclient: _renew_subs 2024-03-25T19:15:36.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.720+0000 7f1f05e9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:15:36.722 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.720+0000 7f1f05e9a640 1 -- 172.21.15.6:0/2771991392 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f0013f280 con 0x7f1f000b4ff0 2024-03-25T19:15:36.722 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.720+0000 7f1f05e9a640 1 librados: init done 2024-03-25T19:15:36.722 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.721+0000 7f1f05e9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:15:36.722 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.721+0000 7f1f05e9a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:15:36.722 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.721+0000 7f1f05e9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:15:36.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:36.721+0000 7f1f05e9a640 1 -- 172.21.15.6:0/2771991392 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f0006dbd0 con 0x7f1f000b4ff0 2024-03-25T19:15:37.034 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:37.033+0000 7f1ee7fff640 1 -- 172.21.15.6:0/2771991392 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ef40038a0 con 0x7f1f000b4ff0 2024-03-25T19:15:37.034 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:37.033+0000 7f1ee7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:15:37.034 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:37.033+0000 7f1ee7fff640 1 -- 172.21.15.6:0/2771991392 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f1ef4003bb0 con 0x7f1f000b4ff0 2024-03-25T19:15:37.034 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:37.033+0000 7f1ee7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:15:37.034 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:37.033+0000 7f1ee7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:15:37.034 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:37.033+0000 7f1efdffb640 4 set_mon_vals no callback set 2024-03-25T19:15:37.034 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:37.033+0000 7f1ee7fff640 10 monclient: dump: 2024-03-25T19:15:37.034 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:15:37.034 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:37.034 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:37.034 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:37.034 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:15:37.035 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:37.035 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:15:37.035 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:15:37.035 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:15:37.035 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:37.035 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:37.034+0000 7f1ee7fff640 1 -- 172.21.15.6:0/2771991392 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f1ef4003dd0 con 0x7f1f000b4ff0 2024-03-25T19:15:37.036 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:37.035+0000 7f1eff7fe640 10 monclient: get_auth_request con 0x7f1eec06e160 auth_method 0 2024-03-25T19:15:37.036 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:37.035+0000 7f1ee7fff640 10 client.4288.objecter ms_handle_connect 0x7f1eec06e160 2024-03-25T19:15:37.039 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:37.038+0000 7f1ee7fff640 1 -- 172.21.15.6:0/2771991392 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f1ef4054860 con 0x7f1f000b4ff0 2024-03-25T19:15:37.039 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:37.038+0000 7f1ee7fff640 10 client.4288.objecter ms_dispatch 0x7f1f00000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:15:37.039 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:37.038+0000 7f1ee7fff640 3 client.4288.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:15:37.039 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:37.038+0000 7f1ee7fff640 3 client.4288.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:15:37.039 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:37.038+0000 7f1ee7fff640 20 client.4288.objecter dump_active .. 0 homeless 2024-03-25T19:15:37.039 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:37.040+0000 7f1ee7fff640 1 -- 172.21.15.6:0/2771991392 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1ef4050370 con 0x7f1f000b4ff0 2024-03-25T19:15:37.039 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:37.040+0000 7f1ee7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:15:37.039 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:37.040+0000 7f1ee7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:15:37.305 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:37.305+0000 7f1f05e9a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:15:37.305 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:37.305+0000 7f1f05e9a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:15:37.305 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:37.305+0000 7f1f05e9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:15:37.305 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:37.305+0000 7f1f05e9a640 1 -- 172.21.15.6:0/2771991392 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f000688c0 con 0x7f1f000b4ff0 2024-03-25T19:15:37.356 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:37.355+0000 7f1ee7fff640 1 -- 172.21.15.6:0/2771991392 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f1ef40522a0 con 0x7f1f000b4ff0 2024-03-25T19:15:37.356 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:37.355+0000 7f1ee7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:15:37.356 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:37.355+0000 7f1ee7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:15:37.356 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:15:37.356 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:15:37.359 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:37.358+0000 7f1f05e9a640 10 librados: watch_flush enter 2024-03-25T19:15:37.359 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:37.358+0000 7f1f05e9a640 10 librados: watch_flush exit 2024-03-25T19:15:37.359 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:37.358+0000 7f1f05e9a640 20 client.4288.objecter shutdown clearing up homeless session... 2024-03-25T19:15:37.359 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:37.358+0000 7f1f05e9a640 10 client.4288.objecter successfully canceled tick 2024-03-25T19:15:37.359 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:37.358+0000 7f1f05e9a640 1 -- 172.21.15.6:0/2771991392 >> v1:172.21.15.6:6800/2936652650 conn(0x7f1eec06e160 legacy=0x7f1eec070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:37.359 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:37.358+0000 7f1f05e9a640 10 monclient: shutdown 2024-03-25T19:15:37.359 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:37.358+0000 7f1f05e9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:15:37.359 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:37.358+0000 7f1f05e9a640 1 -- 172.21.15.6:0/2771991392 >> v1:172.21.15.192:6789/0 conn(0x7f1f000b4ff0 legacy=0x7f1f0012fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:37.360 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:37.358+0000 7f1f05e9a640 1 -- 172.21.15.6:0/2771991392 shutdown_connections 2024-03-25T19:15:37.360 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:37.358+0000 7f1f05e9a640 1 -- 172.21.15.6:0/2771991392 >> 172.21.15.6:0/2771991392 conn(0x7f1f000617a0 msgr2=0x7f1f001407f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:15:37.360 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:37.359+0000 7f1f05e9a640 1 -- 172.21.15.6:0/2771991392 shutdown_connections 2024-03-25T19:15:37.360 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:37.359+0000 7f1f05e9a640 1 -- 172.21.15.6:0/2771991392 wait complete. 2024-03-25T19:15:37.360 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:37.359+0000 7f1f05e9a640 1 librados: shutdown 2024-03-25T19:15:37.371 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:15:42.165 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:01:59.511 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:15:42.174 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:01:59.521 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:15:42.214 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:01:59.561 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:15:42.232 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:01:59.550 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:15:42.237 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:01:59.555 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:15:42.247 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:01:59.594 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:15:42.252 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:01:59.599 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:15:42.267 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:01:59.585 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:15:42.281 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:01:59.628 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:15:44.177 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:15:44.276 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.275+0000 7f9f5c855640 10 monclient: get_monmap_and_config 2024-03-25T19:15:44.276 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.275+0000 7f9f5c855640 10 monclient: build_initial_monmap 2024-03-25T19:15:44.276 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.275+0000 7f9f5c855640 10 monclient: monmap: 2024-03-25T19:15:44.276 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:15:44.276 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:44.276 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:15:44.277306+0000 2024-03-25T19:15:44.276 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:15:44.277306+0000 2024-03-25T19:15:44.276 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:15:44.277 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:44.277 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:15:44.277 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:15:44.277 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:15:44.277 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:44.278 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.275+0000 7f9f5c855640 1 Processor -- start 2024-03-25T19:15:44.278 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.276+0000 7f9f5c855640 1 -- start start 2024-03-25T19:15:44.278 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.276+0000 7f9f5c855640 10 monclient: init 2024-03-25T19:15:44.278 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.276+0000 7f9f5c855640 10 monclient: _reopen_session rank -1 2024-03-25T19:15:44.278 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.276+0000 7f9f5c855640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:15:44.278 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.276+0000 7f9f5c855640 10 monclient(hunting): picked mon.noname-c con 0x7f9f580b4e70 addr v1:172.21.15.192:6789/0 2024-03-25T19:15:44.278 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.277+0000 7f9f5c855640 10 monclient(hunting): picked mon.noname-a con 0x7f9f580b44b0 addr v1:172.21.15.6:6789/0 2024-03-25T19:15:44.278 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.277+0000 7f9f5c855640 10 monclient(hunting): picked mon.noname-b con 0x7f9f580b1b00 addr v1:172.21.15.6:6790/0 2024-03-25T19:15:44.279 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.277+0000 7f9f5c855640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f580a96f0 con 0x7f9f580b44b0 2024-03-25T19:15:44.279 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.277+0000 7f9f5c855640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f580ab0e0 con 0x7f9f580b4e70 2024-03-25T19:15:44.279 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.277+0000 7f9f5c855640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f58145f70 con 0x7f9f580b1b00 2024-03-25T19:15:44.279 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.277+0000 7f9f5c855640 10 monclient(hunting): _renew_subs 2024-03-25T19:15:44.279 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.277+0000 7f9f5c855640 10 monclient(hunting): authenticate will time out at 6048.517860s 2024-03-25T19:15:44.282 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.281+0000 7f9f55d9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9f580b44b0 0x7f9f58145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53760/0 (socket says 172.21.15.6:53760) 2024-03-25T19:15:44.282 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.281+0000 7f9f55d9b640 1 -- 172.21.15.6:0/3542530571 learned_addr learned my addr 172.21.15.6:0/3542530571 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:15:44.298 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.297+0000 7f9f5559a640 1 -- 172.21.15.6:0/3542530571 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3900171328 0 0) 0x7f9f58145f70 con 0x7f9f580b1b00 2024-03-25T19:15:44.298 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.297+0000 7f9f5559a640 10 monclient(hunting): _negotiate 2024-03-25T19:15:44.298 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.297+0000 7f9f5559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:15:44.298 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.297+0000 7f9f5559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:15:44.298 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.297+0000 7f9f5559a640 10 monclient(hunting): my global_id is 4295 2024-03-25T19:15:44.298 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.297+0000 7f9f5559a640 1 -- 172.21.15.6:0/3542530571 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f40003520 con 0x7f9f580b1b00 2024-03-25T19:15:44.302 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.301+0000 7f9f5559a640 1 -- 172.21.15.6:0/3542530571 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 608559419 0 0) 0x7f9f580a96f0 con 0x7f9f580b44b0 2024-03-25T19:15:44.302 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.301+0000 7f9f5559a640 10 monclient(hunting): _negotiate 2024-03-25T19:15:44.302 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.301+0000 7f9f5559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:15:44.303 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.301+0000 7f9f5559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:15:44.303 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.301+0000 7f9f5559a640 10 monclient(hunting): my global_id is 4290 2024-03-25T19:15:44.303 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.301+0000 7f9f5559a640 1 -- 172.21.15.6:0/3542530571 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f58145f70 con 0x7f9f580b44b0 2024-03-25T19:15:44.308 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.307+0000 7f9f5559a640 1 -- 172.21.15.6:0/3542530571 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2468807724 0 0) 0x7f9f40003520 con 0x7f9f580b1b00 2024-03-25T19:15:44.308 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.307+0000 7f9f5559a640 1 -- 172.21.15.6:0/3542530571 >> v1:172.21.15.192:6789/0 conn(0x7f9f580b4e70 legacy=0x7f9f5813f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:44.308 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.307+0000 7f9f5559a640 1 -- 172.21.15.6:0/3542530571 >> v1:172.21.15.6:6789/0 conn(0x7f9f580b44b0 legacy=0x7f9f58145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:44.308 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.307+0000 7f9f5559a640 10 monclient: _finish_hunting 0 2024-03-25T19:15:44.308 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.307+0000 7f9f5559a640 1 monclient: found mon.noname-b 2024-03-25T19:15:44.308 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.307+0000 7f9f5559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:15:44.308 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.307+0000 7f9f5559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-03-25T19:15:44.308 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.307+0000 7f9f5559a640 1 -- 172.21.15.6:0/3542530571 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f580a9b80 con 0x7f9f580b1b00 2024-03-25T19:15:44.309 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.307+0000 7f9f5559a640 10 monclient: _finish_auth 0 2024-03-25T19:15:44.309 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.307+0000 7f9f5559a640 10 monclient: _check_auth_tickets 2024-03-25T19:15:44.309 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.307+0000 7f9f5559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:15:44.309 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.307+0000 7f9f5559a640 1 -- 172.21.15.6:0/3542530571 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f9f50001a10 con 0x7f9f580b1b00 2024-03-25T19:15:44.309 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.307+0000 7f9f5559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:15:44.309 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.307+0000 7f9f5559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:15:44.309 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.307+0000 7f9f5559a640 10 monclient: dump: 2024-03-25T19:15:44.309 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:15:44.309 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:44.309 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:44.309 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:44.309 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:15:44.309 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:44.309 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:15:44.310 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:15:44.310 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:15:44.310 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:44.310 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.307+0000 7f9f5c855640 5 monclient: authenticate success, global_id 4295 2024-03-25T19:15:44.310 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.307+0000 7f9f5c855640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:15:44.325 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.324+0000 7f9f5559a640 1 -- 172.21.15.6:0/3542530571 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f50001b70 con 0x7f9f580b1b00 2024-03-25T19:15:44.325 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.324+0000 7f9f5559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:15:44.325 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.324+0000 7f9f5559a640 1 -- 172.21.15.6:0/3542530571 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f9f50001f80 con 0x7f9f580b1b00 2024-03-25T19:15:44.325 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.324+0000 7f9f5559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:15:44.325 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.324+0000 7f9f5559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:15:44.325 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.324+0000 7f9f5559a640 10 monclient: dump: 2024-03-25T19:15:44.325 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:15:44.325 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:44.325 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:44.326 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:44.326 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:15:44.326 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:44.326 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:15:44.326 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:15:44.326 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:15:44.326 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:44.326 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.324+0000 7f9f5c855640 10 monclient: get_monmap_and_config success 2024-03-25T19:15:44.326 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.324+0000 7f9f5c855640 4 set_mon_vals no callback set 2024-03-25T19:15:44.327 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.325+0000 7f9f5c855640 10 monclient: shutdown 2024-03-25T19:15:44.328 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.325+0000 7f9f5c855640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:15:44.328 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.325+0000 7f9f5c855640 1 -- 172.21.15.6:0/3542530571 >> v1:172.21.15.6:6790/0 conn(0x7f9f580b1b00 legacy=0x7f9f5814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:44.328 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.325+0000 7f9f5c855640 1 -- 172.21.15.6:0/3542530571 shutdown_connections 2024-03-25T19:15:44.328 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.325+0000 7f9f5c855640 1 -- 172.21.15.6:0/3542530571 >> 172.21.15.6:0/3542530571 conn(0x7f9f580617a0 msgr2=0x7f9f58061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:15:44.328 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.325+0000 7f9f5c855640 1 -- 172.21.15.6:0/3542530571 shutdown_connections 2024-03-25T19:15:44.328 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.325+0000 7f9f5c855640 1 -- 172.21.15.6:0/3542530571 wait complete. 2024-03-25T19:15:44.328 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.326+0000 7f9f5c855640 10 monclient: build_initial_monmap 2024-03-25T19:15:44.328 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.326+0000 7f9f5c855640 10 monclient: monmap: 2024-03-25T19:15:44.328 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:15:44.328 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:15:44.328 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:15:44.328 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:15:44.329 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:15:44.329 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:15:44.329 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:15:44.329 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:15:44.329 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:15:44.329 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:44.329 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.326+0000 7f9f5c855640 1 librados: starting msgr at 2024-03-25T19:15:44.329 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.326+0000 7f9f5c855640 1 librados: starting objecter 2024-03-25T19:15:44.329 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.326+0000 7f9f5c855640 1 Processor -- start 2024-03-25T19:15:44.329 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.327+0000 7f9f5c855640 1 -- start start 2024-03-25T19:15:44.329 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.327+0000 7f9f5c855640 1 librados: setting wanted keys 2024-03-25T19:15:44.329 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.327+0000 7f9f5c855640 1 librados: calling monclient init 2024-03-25T19:15:44.329 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.327+0000 7f9f5c855640 10 monclient: init 2024-03-25T19:15:44.330 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.327+0000 7f9f5c855640 10 monclient: _reopen_session rank -1 2024-03-25T19:15:44.331 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.327+0000 7f9f5c855640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:15:44.331 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.327+0000 7f9f5c855640 10 monclient(hunting): picked mon.noname-a con 0x7f9f580b1b00 addr v1:172.21.15.6:6789/0 2024-03-25T19:15:44.331 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.327+0000 7f9f5c855640 10 monclient(hunting): picked mon.noname-b con 0x7f9f580b44b0 addr v1:172.21.15.192:6789/0 2024-03-25T19:15:44.331 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.327+0000 7f9f5c855640 10 monclient(hunting): picked mon.noname-c con 0x7f9f580b4e70 addr v1:172.21.15.6:6790/0 2024-03-25T19:15:44.331 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.327+0000 7f9f5c855640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f5813b200 con 0x7f9f580b1b00 2024-03-25T19:15:44.331 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.327+0000 7f9f5c855640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f5813c040 con 0x7f9f580b44b0 2024-03-25T19:15:44.331 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.327+0000 7f9f5c855640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f5813af20 con 0x7f9f580b4e70 2024-03-25T19:15:44.331 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.327+0000 7f9f5c855640 10 monclient(hunting): _renew_subs 2024-03-25T19:15:44.331 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.327+0000 7f9f5c855640 10 monclient(hunting): authenticate will time out at 6048.568564s 2024-03-25T19:15:44.334 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.334+0000 7f9f55d9b640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f9f580b44b0 0x7f9f581d4c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:47682/0 (socket says 172.21.15.6:47682) 2024-03-25T19:15:44.334 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.334+0000 7f9f55d9b640 1 -- 172.21.15.6:0/3143462399 learned_addr learned my addr 172.21.15.6:0/3143462399 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:15:44.338 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.337+0000 7f9f46ffd640 10 client.?.objecter ms_handle_connect 0x7f9f580b44b0 2024-03-25T19:15:44.338 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.337+0000 7f9f46ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:15:44.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.342+0000 7f9f46ffd640 10 client.?.objecter ms_handle_connect 0x7f9f580b1b00 2024-03-25T19:15:44.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.342+0000 7f9f46ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:15:44.353 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.352+0000 7f9f46ffd640 1 -- 172.21.15.6:0/3143462399 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3921018139 0 0) 0x7f9f5813c040 con 0x7f9f580b44b0 2024-03-25T19:15:44.353 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.352+0000 7f9f46ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:15:44.353 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.352+0000 7f9f46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:15:44.353 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.352+0000 7f9f46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:15:44.353 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.352+0000 7f9f46ffd640 10 monclient(hunting): my global_id is 4294 2024-03-25T19:15:44.353 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.352+0000 7f9f46ffd640 1 -- 172.21.15.6:0/3143462399 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f38004eb0 con 0x7f9f580b44b0 2024-03-25T19:15:44.355 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.354+0000 7f9f46ffd640 1 -- 172.21.15.6:0/3143462399 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4270476024 0 0) 0x7f9f5813b200 con 0x7f9f580b1b00 2024-03-25T19:15:44.355 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.354+0000 7f9f46ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:15:44.355 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.354+0000 7f9f46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:15:44.355 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.354+0000 7f9f46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:15:44.355 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.354+0000 7f9f46ffd640 10 monclient(hunting): my global_id is 4293 2024-03-25T19:15:44.355 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.354+0000 7f9f46ffd640 1 -- 172.21.15.6:0/3143462399 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f5813c040 con 0x7f9f580b1b00 2024-03-25T19:15:44.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.361+0000 7f9f46ffd640 10 client.?.objecter ms_handle_connect 0x7f9f580b4e70 2024-03-25T19:15:44.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.361+0000 7f9f46ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:15:44.372 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.371+0000 7f9f46ffd640 1 -- 172.21.15.6:0/3143462399 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3352314387 0 0) 0x7f9f5813c040 con 0x7f9f580b1b00 2024-03-25T19:15:44.372 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.371+0000 7f9f46ffd640 1 -- 172.21.15.6:0/3143462399 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f5813b200 con 0x7f9f580b1b00 2024-03-25T19:15:44.372 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.371+0000 7f9f46ffd640 1 -- 172.21.15.6:0/3143462399 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f9f48001890 con 0x7f9f580b1b00 2024-03-25T19:15:44.373 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.371+0000 7f9f46ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:15:44.373 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.371+0000 7f9f46ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:15:44.373 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.371+0000 7f9f46ffd640 10 monclient(hunting): dump: 2024-03-25T19:15:44.373 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:15:44.373 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:44.373 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:44.373 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:44.373 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:15:44.373 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:44.373 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:15:44.373 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:15:44.373 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:15:44.373 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:44.373 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.371+0000 7f9f46ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:15:44.373 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.371+0000 7f9f46ffd640 1 -- 172.21.15.6:0/3143462399 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1310343176 0 0) 0x7f9f38004eb0 con 0x7f9f580b44b0 2024-03-25T19:15:44.374 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.371+0000 7f9f46ffd640 1 -- 172.21.15.6:0/3143462399 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f5813c040 con 0x7f9f580b44b0 2024-03-25T19:15:44.374 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.371+0000 7f9f46ffd640 1 -- 172.21.15.6:0/3143462399 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f9f4c001970 con 0x7f9f580b44b0 2024-03-25T19:15:44.374 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.371+0000 7f9f46ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:15:44.374 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.371+0000 7f9f46ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:15:44.374 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.371+0000 7f9f46ffd640 10 monclient(hunting): dump: 2024-03-25T19:15:44.374 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:15:44.374 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:44.374 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:44.374 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:44.374 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:15:44.374 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:44.374 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:15:44.374 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:15:44.375 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:15:44.375 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:44.379 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.377+0000 7f9f46ffd640 1 -- 172.21.15.6:0/3143462399 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2493674437 0 0) 0x7f9f5813b200 con 0x7f9f580b1b00 2024-03-25T19:15:44.379 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.377+0000 7f9f46ffd640 1 -- 172.21.15.6:0/3143462399 >> v1:172.21.15.6:6790/0 conn(0x7f9f580b4e70 legacy=0x7f9f581e2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:44.379 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.377+0000 7f9f46ffd640 1 -- 172.21.15.6:0/3143462399 >> v1:172.21.15.192:6789/0 conn(0x7f9f580b44b0 legacy=0x7f9f581d4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:44.379 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.377+0000 7f9f46ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:15:44.380 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.377+0000 7f9f46ffd640 1 monclient: found mon.a 2024-03-25T19:15:44.380 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.377+0000 7f9f46ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:15:44.380 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.377+0000 7f9f46ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:15:44.380 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.377+0000 7f9f46ffd640 1 -- 172.21.15.6:0/3143462399 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f5813c290 con 0x7f9f580b1b00 2024-03-25T19:15:44.380 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.377+0000 7f9f46ffd640 10 monclient: _finish_auth 0 2024-03-25T19:15:44.380 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.377+0000 7f9f46ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:15:44.380 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.377+0000 7f9f46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:15:44.380 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.377+0000 7f9f5c855640 5 monclient: authenticate success, global_id 4293 2024-03-25T19:15:44.380 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.377+0000 7f9f5c855640 10 monclient: _renew_subs 2024-03-25T19:15:44.380 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.377+0000 7f9f5c855640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:15:44.380 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.377+0000 7f9f5c855640 1 -- 172.21.15.6:0/3143462399 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f5813ca70 con 0x7f9f580b1b00 2024-03-25T19:15:44.380 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.377+0000 7f9f5c855640 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:15:44.380 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.377+0000 7f9f5c855640 10 monclient: _renew_subs 2024-03-25T19:15:44.380 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.377+0000 7f9f5c855640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:15:44.381 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.377+0000 7f9f5c855640 1 -- 172.21.15.6:0/3143462399 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f5813a8a0 con 0x7f9f580b1b00 2024-03-25T19:15:44.381 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.377+0000 7f9f5c855640 1 librados: init done 2024-03-25T19:15:44.381 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.378+0000 7f9f5c855640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:15:44.381 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.378+0000 7f9f5c855640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:15:44.381 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.378+0000 7f9f5c855640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:15:44.381 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.378+0000 7f9f5c855640 1 -- 172.21.15.6:0/3143462399 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f5813c7b0 con 0x7f9f580b1b00 2024-03-25T19:15:44.704 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.703+0000 7f9f46ffd640 1 -- 172.21.15.6:0/3143462399 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f48001d40 con 0x7f9f580b1b00 2024-03-25T19:15:44.704 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.703+0000 7f9f46ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:15:44.704 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.703+0000 7f9f46ffd640 1 -- 172.21.15.6:0/3143462399 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f9f48003610 con 0x7f9f580b1b00 2024-03-25T19:15:44.704 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.703+0000 7f9f46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:15:44.704 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.703+0000 7f9f46ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:15:44.704 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.703+0000 7f9f46ffd640 10 monclient: dump: 2024-03-25T19:15:44.704 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:15:44.705 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:44.705 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:44.705 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:44.705 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:15:44.705 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:44.705 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:15:44.705 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:15:44.705 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:15:44.705 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:44.705 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.703+0000 7f9f54d99640 4 set_mon_vals no callback set 2024-03-25T19:15:44.705 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.703+0000 7f9f46ffd640 1 -- 172.21.15.6:0/3143462399 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f9f48003830 con 0x7f9f580b1b00 2024-03-25T19:15:44.706 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.704+0000 7f9f55d9b640 10 monclient: get_auth_request con 0x7f9f3806de40 auth_method 0 2024-03-25T19:15:44.706 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.704+0000 7f9f46ffd640 10 client.4293.objecter ms_handle_connect 0x7f9f3806de40 2024-03-25T19:15:44.710 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.709+0000 7f9f46ffd640 1 -- 172.21.15.6:0/3143462399 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f9f480533f0 con 0x7f9f580b1b00 2024-03-25T19:15:44.710 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.709+0000 7f9f46ffd640 10 client.4293.objecter ms_dispatch 0x7f9f58000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:15:44.710 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.709+0000 7f9f46ffd640 3 client.4293.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:15:44.710 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.709+0000 7f9f46ffd640 3 client.4293.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:15:44.710 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.709+0000 7f9f46ffd640 20 client.4293.objecter dump_active .. 0 homeless 2024-03-25T19:15:44.711 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.709+0000 7f9f46ffd640 1 -- 172.21.15.6:0/3143462399 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9f4804ed30 con 0x7f9f580b1b00 2024-03-25T19:15:44.711 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.709+0000 7f9f46ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:15:44.711 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.709+0000 7f9f46ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:15:44.971 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.970+0000 7f9f5c855640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:15:44.971 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.970+0000 7f9f5c855640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:15:44.971 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.970+0000 7f9f5c855640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:15:44.971 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.970+0000 7f9f5c855640 1 -- 172.21.15.6:0/3143462399 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f5804f730 con 0x7f9f580b1b00 2024-03-25T19:15:44.996 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.995+0000 7f9f46ffd640 1 -- 172.21.15.6:0/3143462399 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f9f48050e30 con 0x7f9f580b1b00 2024-03-25T19:15:44.996 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.995+0000 7f9f46ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:15:44.996 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.995+0000 7f9f46ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:15:44.996 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:15:44.996 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:15:44.998 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.997+0000 7f9f5c855640 10 librados: watch_flush enter 2024-03-25T19:15:44.999 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.997+0000 7f9f5c855640 10 librados: watch_flush exit 2024-03-25T19:15:44.999 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.997+0000 7f9f5c855640 20 client.4293.objecter shutdown clearing up homeless session... 2024-03-25T19:15:44.999 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.997+0000 7f9f5c855640 10 client.4293.objecter successfully canceled tick 2024-03-25T19:15:44.999 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.997+0000 7f9f5c855640 1 -- 172.21.15.6:0/3143462399 >> v1:172.21.15.6:6800/2936652650 conn(0x7f9f3806de40 legacy=0x7f9f38070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:44.999 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.997+0000 7f9f5c855640 10 monclient: shutdown 2024-03-25T19:15:44.999 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.997+0000 7f9f5c855640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:15:44.999 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.997+0000 7f9f5c855640 1 -- 172.21.15.6:0/3143462399 >> v1:172.21.15.6:6789/0 conn(0x7f9f580b1b00 legacy=0x7f9f581d4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:44.999 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.997+0000 7f9f5c855640 1 -- 172.21.15.6:0/3143462399 shutdown_connections 2024-03-25T19:15:44.999 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.997+0000 7f9f5c855640 1 -- 172.21.15.6:0/3143462399 >> 172.21.15.6:0/3143462399 conn(0x7f9f580617a0 msgr2=0x7f9f580b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:15:44.999 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.997+0000 7f9f5c855640 1 -- 172.21.15.6:0/3143462399 shutdown_connections 2024-03-25T19:15:44.999 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.997+0000 7f9f5c855640 1 -- 172.21.15.6:0/3143462399 wait complete. 2024-03-25T19:15:44.999 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:44.997+0000 7f9f5c855640 1 librados: shutdown 2024-03-25T19:15:45.010 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:15:51.817 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:15:51.933 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.933+0000 7f6da3086640 10 monclient: get_monmap_and_config 2024-03-25T19:15:51.933 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.933+0000 7f6da3086640 10 monclient: build_initial_monmap 2024-03-25T19:15:51.933 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.933+0000 7f6da3086640 10 monclient: monmap: 2024-03-25T19:15:51.933 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:15:51.933 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:51.933 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:15:51.934418+0000 2024-03-25T19:15:51.933 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:15:51.934418+0000 2024-03-25T19:15:51.933 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:15:51.934 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:51.934 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:15:51.934 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:15:51.934 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:15:51.934 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:51.935 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.934+0000 7f6da3086640 1 Processor -- start 2024-03-25T19:15:51.935 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.934+0000 7f6da3086640 1 -- start start 2024-03-25T19:15:51.935 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.934+0000 7f6da3086640 10 monclient: init 2024-03-25T19:15:51.935 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.935+0000 7f6da3086640 10 monclient: _reopen_session rank -1 2024-03-25T19:15:51.935 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.935+0000 7f6da3086640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:15:51.936 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.935+0000 7f6da3086640 10 monclient(hunting): picked mon.noname-a con 0x7f6d9c0b1a70 addr v1:172.21.15.6:6789/0 2024-03-25T19:15:51.936 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.935+0000 7f6da3086640 10 monclient(hunting): picked mon.noname-c con 0x7f6d9c146490 addr v1:172.21.15.192:6789/0 2024-03-25T19:15:51.936 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.935+0000 7f6da3086640 10 monclient(hunting): picked mon.noname-b con 0x7f6d9c1458a0 addr v1:172.21.15.6:6790/0 2024-03-25T19:15:51.936 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.935+0000 7f6da3086640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9c145d30 con 0x7f6d9c0b1a70 2024-03-25T19:15:51.936 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.935+0000 7f6da3086640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9c13ffb0 con 0x7f6d9c146490 2024-03-25T19:15:51.936 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.935+0000 7f6da3086640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9c0b26a0 con 0x7f6d9c1458a0 2024-03-25T19:15:51.936 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.935+0000 7f6da3086640 10 monclient(hunting): _renew_subs 2024-03-25T19:15:51.936 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.935+0000 7f6da3086640 10 monclient(hunting): authenticate will time out at 6056.175093s 2024-03-25T19:15:51.946 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.946+0000 7f6d93fff640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f6d9c146490 0x7f6d9c0afd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:49106/0 (socket says 172.21.15.6:49106) 2024-03-25T19:15:51.946 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.946+0000 7f6d93fff640 1 -- 172.21.15.6:0/3009686054 learned_addr learned my addr 172.21.15.6:0/3009686054 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:15:51.965 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.966+0000 7f6d937fe640 1 -- 172.21.15.6:0/3009686054 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2799446529 0 0) 0x7f6d9c13ffb0 con 0x7f6d9c146490 2024-03-25T19:15:51.965 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.966+0000 7f6d937fe640 10 monclient(hunting): _negotiate 2024-03-25T19:15:51.966 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.966+0000 7f6d937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:15:51.966 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.966+0000 7f6d937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:15:51.966 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.966+0000 7f6d937fe640 10 monclient(hunting): my global_id is 4297 2024-03-25T19:15:51.966 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.966+0000 7f6d937fe640 1 -- 172.21.15.6:0/3009686054 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d84003520 con 0x7f6d9c146490 2024-03-25T19:15:51.967 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.967+0000 7f6d937fe640 1 -- 172.21.15.6:0/3009686054 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1738684845 0 0) 0x7f6d98001110 con 0x7f6d9c0b1a70 2024-03-25T19:15:51.967 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.967+0000 7f6d937fe640 10 monclient(hunting): _negotiate 2024-03-25T19:15:51.967 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.967+0000 7f6d937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:15:51.967 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.967+0000 7f6d937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:15:51.968 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.967+0000 7f6d937fe640 10 monclient(hunting): my global_id is 4296 2024-03-25T19:15:51.968 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.967+0000 7f6d937fe640 1 -- 172.21.15.6:0/3009686054 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d9c13ffb0 con 0x7f6d9c0b1a70 2024-03-25T19:15:51.977 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.977+0000 7f6d937fe640 1 -- 172.21.15.6:0/3009686054 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 45680212 0 0) 0x7f6d84003520 con 0x7f6d9c146490 2024-03-25T19:15:51.977 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.977+0000 7f6d937fe640 1 -- 172.21.15.6:0/3009686054 >> v1:172.21.15.6:6790/0 conn(0x7f6d9c1458a0 legacy=0x7f6d9c14dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:51.977 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.977+0000 7f6d937fe640 1 -- 172.21.15.6:0/3009686054 >> v1:172.21.15.6:6789/0 conn(0x7f6d9c0b1a70 legacy=0x7f6d9c06bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:51.977 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.977+0000 7f6d937fe640 10 monclient: _finish_hunting 0 2024-03-25T19:15:51.977 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.977+0000 7f6d937fe640 1 monclient: found mon.noname-c 2024-03-25T19:15:51.977 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.977+0000 7f6d937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:15:51.977 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.977+0000 7f6d937fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:15:51.977 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.977+0000 7f6d937fe640 1 -- 172.21.15.6:0/3009686054 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d9c0ae310 con 0x7f6d9c146490 2024-03-25T19:15:51.978 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.977+0000 7f6d937fe640 10 monclient: _finish_auth 0 2024-03-25T19:15:51.978 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.977+0000 7f6d937fe640 10 monclient: _check_auth_tickets 2024-03-25T19:15:51.978 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.977+0000 7f6d937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:15:51.978 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.977+0000 7f6da3086640 5 monclient: authenticate success, global_id 4297 2024-03-25T19:15:51.978 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.977+0000 7f6da3086640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:15:51.982 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.982+0000 7f6d937fe640 1 -- 172.21.15.6:0/3009686054 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f6d8c001650 con 0x7f6d9c146490 2024-03-25T19:15:51.982 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.982+0000 7f6d937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:15:51.982 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.982+0000 7f6d937fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:15:51.982 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.982+0000 7f6d937fe640 10 monclient: dump: 2024-03-25T19:15:51.983 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:15:51.983 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:51.983 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:51.983 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:51.983 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:15:51.983 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:51.983 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:15:51.983 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:15:51.983 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:15:51.983 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:51.983 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.982+0000 7f6d937fe640 1 -- 172.21.15.6:0/3009686054 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d8c0018a0 con 0x7f6d9c146490 2024-03-25T19:15:51.983 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.982+0000 7f6da3086640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:15:51.983 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.982+0000 7f6d937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:15:51.983 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.982+0000 7f6da3086640 10 monclient: get_monmap_and_config success 2024-03-25T19:15:51.984 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.982+0000 7f6d937fe640 1 -- 172.21.15.6:0/3009686054 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f6d8c001c30 con 0x7f6d9c146490 2024-03-25T19:15:51.984 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.982+0000 7f6da3086640 4 set_mon_vals no callback set 2024-03-25T19:15:51.984 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.982+0000 7f6d937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:15:51.984 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.982+0000 7f6d937fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:15:51.984 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.982+0000 7f6d937fe640 10 monclient: dump: 2024-03-25T19:15:51.984 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:15:51.984 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:51.984 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:51.984 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:51.984 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:15:51.984 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:51.984 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:15:51.984 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:15:51.984 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:15:51.984 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:51.987 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.983+0000 7f6da3086640 10 monclient: shutdown 2024-03-25T19:15:51.987 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.983+0000 7f6da3086640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:15:51.987 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.983+0000 7f6da3086640 1 -- 172.21.15.6:0/3009686054 >> v1:172.21.15.192:6789/0 conn(0x7f6d9c146490 legacy=0x7f6d9c0afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:51.987 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.983+0000 7f6da3086640 1 -- 172.21.15.6:0/3009686054 shutdown_connections 2024-03-25T19:15:51.987 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.983+0000 7f6da3086640 1 -- 172.21.15.6:0/3009686054 >> 172.21.15.6:0/3009686054 conn(0x7f6d9c0b28f0 msgr2=0x7f6d9c0b2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:15:51.987 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.983+0000 7f6da3086640 1 -- 172.21.15.6:0/3009686054 shutdown_connections 2024-03-25T19:15:51.987 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.983+0000 7f6da3086640 1 -- 172.21.15.6:0/3009686054 wait complete. 2024-03-25T19:15:51.987 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.984+0000 7f6da3086640 10 monclient: build_initial_monmap 2024-03-25T19:15:51.987 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.984+0000 7f6da3086640 10 monclient: monmap: 2024-03-25T19:15:51.987 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:15:51.987 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:15:51.987 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:15:51.987 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:15:51.987 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:15:51.988 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:15:51.988 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:15:51.988 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:15:51.988 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:15:51.988 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:51.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.984+0000 7f6da3086640 1 librados: starting msgr at 2024-03-25T19:15:51.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.984+0000 7f6da3086640 1 librados: starting objecter 2024-03-25T19:15:51.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.985+0000 7f6da3086640 1 Processor -- start 2024-03-25T19:15:51.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.985+0000 7f6da3086640 1 -- start start 2024-03-25T19:15:51.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.985+0000 7f6da3086640 1 librados: setting wanted keys 2024-03-25T19:15:51.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.985+0000 7f6da3086640 1 librados: calling monclient init 2024-03-25T19:15:51.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.985+0000 7f6da3086640 10 monclient: init 2024-03-25T19:15:51.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.986+0000 7f6da3086640 10 monclient: _reopen_session rank -1 2024-03-25T19:15:51.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.986+0000 7f6da3086640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:15:51.989 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.986+0000 7f6da3086640 10 monclient(hunting): picked mon.noname-a con 0x7f6d9c0b1a70 addr v1:172.21.15.6:6789/0 2024-03-25T19:15:51.989 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.986+0000 7f6da3086640 10 monclient(hunting): picked mon.noname-b con 0x7f6d9c1458a0 addr v1:172.21.15.192:6789/0 2024-03-25T19:15:51.989 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.986+0000 7f6da3086640 10 monclient(hunting): picked mon.noname-c con 0x7f6d9c146490 addr v1:172.21.15.6:6790/0 2024-03-25T19:15:51.989 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.986+0000 7f6da3086640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9c140a20 con 0x7f6d9c0b1a70 2024-03-25T19:15:51.989 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.986+0000 7f6da3086640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9c1430f0 con 0x7f6d9c1458a0 2024-03-25T19:15:51.989 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.986+0000 7f6da3086640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9c140c70 con 0x7f6d9c146490 2024-03-25T19:15:51.989 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.986+0000 7f6da3086640 10 monclient(hunting): _renew_subs 2024-03-25T19:15:51.989 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.986+0000 7f6da3086640 10 monclient(hunting): authenticate will time out at 6056.226227s 2024-03-25T19:15:51.996 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.996+0000 7f6da0e22640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6d9c0b1a70 0x7f6d9c13a170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33840/0 (socket says 172.21.15.6:33840) 2024-03-25T19:15:51.996 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:51.996+0000 7f6da0e22640 1 -- 172.21.15.6:0/4209269091 learned_addr learned my addr 172.21.15.6:0/4209269091 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:15:52.000 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.001+0000 7f6d917fa640 10 client.?.objecter ms_handle_connect 0x7f6d9c0b1a70 2024-03-25T19:15:52.001 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.001+0000 7f6d917fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:15:52.004 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.004+0000 7f6d917fa640 10 client.?.objecter ms_handle_connect 0x7f6d9c1458a0 2024-03-25T19:15:52.004 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.004+0000 7f6d917fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:15:52.014 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.014+0000 7f6d917fa640 1 -- 172.21.15.6:0/4209269091 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1691612708 0 0) 0x7f6d9c140a20 con 0x7f6d9c0b1a70 2024-03-25T19:15:52.014 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.014+0000 7f6d917fa640 10 monclient(hunting): _negotiate 2024-03-25T19:15:52.014 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.014+0000 7f6d917fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:15:52.015 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.014+0000 7f6d917fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:15:52.015 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.014+0000 7f6d917fa640 10 monclient(hunting): my global_id is 4299 2024-03-25T19:15:52.015 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.014+0000 7f6d917fa640 1 -- 172.21.15.6:0/4209269091 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d88004eb0 con 0x7f6d9c0b1a70 2024-03-25T19:15:52.020 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.020+0000 7f6d917fa640 10 client.?.objecter ms_handle_connect 0x7f6d9c146490 2024-03-25T19:15:52.020 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.020+0000 7f6d917fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:15:52.021 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.021+0000 7f6d917fa640 1 -- 172.21.15.6:0/4209269091 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 758862455 0 0) 0x7f6d9c1430f0 con 0x7f6d9c1458a0 2024-03-25T19:15:52.021 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.021+0000 7f6d917fa640 10 monclient(hunting): _negotiate 2024-03-25T19:15:52.021 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.021+0000 7f6d917fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:15:52.021 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.021+0000 7f6d917fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:15:52.021 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.021+0000 7f6d917fa640 10 monclient(hunting): my global_id is 4300 2024-03-25T19:15:52.021 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.021+0000 7f6d917fa640 1 -- 172.21.15.6:0/4209269091 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d9c140a20 con 0x7f6d9c1458a0 2024-03-25T19:15:52.028 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.028+0000 7f6d917fa640 1 -- 172.21.15.6:0/4209269091 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3004106190 0 0) 0x7f6d88004eb0 con 0x7f6d9c0b1a70 2024-03-25T19:15:52.028 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.028+0000 7f6d917fa640 1 -- 172.21.15.6:0/4209269091 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d9c1430f0 con 0x7f6d9c0b1a70 2024-03-25T19:15:52.028 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.028+0000 7f6d917fa640 1 -- 172.21.15.6:0/4209269091 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f6d98001920 con 0x7f6d9c0b1a70 2024-03-25T19:15:52.028 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.028+0000 7f6d917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:15:52.028 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.028+0000 7f6d917fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:15:52.028 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.028+0000 7f6d917fa640 10 monclient(hunting): dump: 2024-03-25T19:15:52.028 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:15:52.028 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:52.029 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:52.029 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:52.029 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:15:52.029 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:52.029 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:15:52.029 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:15:52.029 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:15:52.029 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:52.029 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.028+0000 7f6d917fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:15:52.030 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.030+0000 7f6d917fa640 1 -- 172.21.15.6:0/4209269091 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 487510073 0 0) 0x7f6d9c140c70 con 0x7f6d9c146490 2024-03-25T19:15:52.030 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.030+0000 7f6d917fa640 10 monclient(hunting): _negotiate 2024-03-25T19:15:52.030 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.030+0000 7f6d917fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:15:52.030 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.030+0000 7f6d917fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:15:52.030 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.030+0000 7f6d917fa640 10 monclient(hunting): my global_id is 4304 2024-03-25T19:15:52.030 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.030+0000 7f6d917fa640 1 -- 172.21.15.6:0/4209269091 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d88004eb0 con 0x7f6d9c146490 2024-03-25T19:15:52.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.040+0000 7f6d917fa640 1 -- 172.21.15.6:0/4209269091 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1917021094 0 0) 0x7f6d9c1430f0 con 0x7f6d9c0b1a70 2024-03-25T19:15:52.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.040+0000 7f6d917fa640 1 -- 172.21.15.6:0/4209269091 >> v1:172.21.15.6:6790/0 conn(0x7f6d9c146490 legacy=0x7f6d9c1418a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:52.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.040+0000 7f6d917fa640 1 -- 172.21.15.6:0/4209269091 >> v1:172.21.15.192:6789/0 conn(0x7f6d9c1458a0 legacy=0x7f6d9c141190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:52.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.040+0000 7f6d917fa640 10 monclient: _finish_hunting 0 2024-03-25T19:15:52.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.040+0000 7f6d917fa640 1 monclient: found mon.a 2024-03-25T19:15:52.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.040+0000 7f6d917fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:15:52.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.040+0000 7f6d917fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:15:52.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.040+0000 7f6d917fa640 1 -- 172.21.15.6:0/4209269091 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d9c1d5390 con 0x7f6d9c0b1a70 2024-03-25T19:15:52.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.040+0000 7f6d917fa640 10 monclient: _finish_auth 0 2024-03-25T19:15:52.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.040+0000 7f6d917fa640 10 monclient: _check_auth_tickets 2024-03-25T19:15:52.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.040+0000 7f6d917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:15:52.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.040+0000 7f6da3086640 5 monclient: authenticate success, global_id 4299 2024-03-25T19:15:52.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.040+0000 7f6da3086640 10 monclient: _renew_subs 2024-03-25T19:15:52.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.040+0000 7f6da3086640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:15:52.042 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.040+0000 7f6da3086640 1 -- 172.21.15.6:0/4209269091 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d9c1dba60 con 0x7f6d9c0b1a70 2024-03-25T19:15:52.042 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.040+0000 7f6da3086640 10 client.4299.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:15:52.042 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.040+0000 7f6da3086640 10 monclient: _renew_subs 2024-03-25T19:15:52.042 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.040+0000 7f6da3086640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:15:52.042 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.040+0000 7f6da3086640 1 -- 172.21.15.6:0/4209269091 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d9c13aa60 con 0x7f6d9c0b1a70 2024-03-25T19:15:52.042 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.040+0000 7f6da3086640 1 librados: init done 2024-03-25T19:15:52.042 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.041+0000 7f6da3086640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:15:52.042 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.041+0000 7f6da3086640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:15:52.043 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.041+0000 7f6da3086640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:15:52.043 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.041+0000 7f6da3086640 1 -- 172.21.15.6:0/4209269091 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d9c1dbc50 con 0x7f6d9c0b1a70 2024-03-25T19:15:52.364 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.364+0000 7f6d917fa640 1 -- 172.21.15.6:0/4209269091 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d98001d70 con 0x7f6d9c0b1a70 2024-03-25T19:15:52.364 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.364+0000 7f6d917fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:15:52.365 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.364+0000 7f6d917fa640 1 -- 172.21.15.6:0/4209269091 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f6d980037b0 con 0x7f6d9c0b1a70 2024-03-25T19:15:52.365 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.364+0000 7f6d917fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:15:52.365 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.364+0000 7f6d917fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:15:52.365 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.364+0000 7f6d917fa640 10 monclient: dump: 2024-03-25T19:15:52.365 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:15:52.365 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:52.365 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:52.365 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:52.365 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:15:52.365 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:52.365 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:15:52.365 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:15:52.365 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:15:52.365 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:52.366 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.364+0000 7f6d92ffd640 4 set_mon_vals no callback set 2024-03-25T19:15:52.366 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.365+0000 7f6d917fa640 1 -- 172.21.15.6:0/4209269091 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f6d980039d0 con 0x7f6d9c0b1a70 2024-03-25T19:15:52.366 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.366+0000 7f6d93fff640 10 monclient: get_auth_request con 0x7f6d8806ddb0 auth_method 0 2024-03-25T19:15:52.366 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.366+0000 7f6d917fa640 10 client.4299.objecter ms_handle_connect 0x7f6d8806ddb0 2024-03-25T19:15:52.375 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.375+0000 7f6d917fa640 1 -- 172.21.15.6:0/4209269091 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f6d98054750 con 0x7f6d9c0b1a70 2024-03-25T19:15:52.375 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.375+0000 7f6d917fa640 10 client.4299.objecter ms_dispatch 0x7f6d9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:15:52.375 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.375+0000 7f6d917fa640 3 client.4299.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:15:52.375 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.375+0000 7f6d917fa640 3 client.4299.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:15:52.375 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.375+0000 7f6d917fa640 20 client.4299.objecter dump_active .. 0 homeless 2024-03-25T19:15:52.375 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.375+0000 7f6d917fa640 1 -- 172.21.15.6:0/4209269091 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6d98050090 con 0x7f6d9c0b1a70 2024-03-25T19:15:52.375 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.375+0000 7f6d917fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:15:52.375 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.375+0000 7f6d917fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:15:52.643 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.643+0000 7f6da3086640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:15:52.643 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.643+0000 7f6da3086640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:15:52.643 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.643+0000 7f6da3086640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:15:52.643 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.643+0000 7f6da3086640 1 -- 172.21.15.6:0/4209269091 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d9c140e60 con 0x7f6d9c0b1a70 2024-03-25T19:15:52.686 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.686+0000 7f6d917fa640 1 -- 172.21.15.6:0/4209269091 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f6d98052190 con 0x7f6d9c0b1a70 2024-03-25T19:15:52.686 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.686+0000 7f6d917fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:15:52.687 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.686+0000 7f6d917fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:15:52.687 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:15:52.687 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:15:52.690 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.689+0000 7f6da3086640 10 librados: watch_flush enter 2024-03-25T19:15:52.690 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.689+0000 7f6da3086640 10 librados: watch_flush exit 2024-03-25T19:15:52.690 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.689+0000 7f6da3086640 20 client.4299.objecter shutdown clearing up homeless session... 2024-03-25T19:15:52.690 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.689+0000 7f6da3086640 10 client.4299.objecter successfully canceled tick 2024-03-25T19:15:52.690 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.689+0000 7f6da3086640 1 -- 172.21.15.6:0/4209269091 >> v1:172.21.15.6:6800/2936652650 conn(0x7f6d8806ddb0 legacy=0x7f6d88070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:52.690 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.689+0000 7f6da3086640 10 monclient: shutdown 2024-03-25T19:15:52.690 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.689+0000 7f6da3086640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:15:52.690 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.689+0000 7f6da3086640 1 -- 172.21.15.6:0/4209269091 >> v1:172.21.15.6:6789/0 conn(0x7f6d9c0b1a70 legacy=0x7f6d9c13a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:52.690 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.689+0000 7f6da3086640 1 -- 172.21.15.6:0/4209269091 shutdown_connections 2024-03-25T19:15:52.690 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.689+0000 7f6da3086640 1 -- 172.21.15.6:0/4209269091 >> 172.21.15.6:0/4209269091 conn(0x7f6d9c0b28f0 msgr2=0x7f6d9c06bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:15:52.690 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.690+0000 7f6da3086640 1 -- 172.21.15.6:0/4209269091 shutdown_connections 2024-03-25T19:15:52.690 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.690+0000 7f6da3086640 1 -- 172.21.15.6:0/4209269091 wait complete. 2024-03-25T19:15:52.691 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:52.690+0000 7f6da3086640 1 librados: shutdown 2024-03-25T19:15:52.702 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:15:59.509 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:15:59.624 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.624+0000 7fd6d220d640 10 monclient: get_monmap_and_config 2024-03-25T19:15:59.624 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.624+0000 7fd6d220d640 10 monclient: build_initial_monmap 2024-03-25T19:15:59.624 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.624+0000 7fd6d220d640 10 monclient: monmap: 2024-03-25T19:15:59.625 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:15:59.625 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:59.625 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:15:59.625668+0000 2024-03-25T19:15:59.625 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:15:59.625668+0000 2024-03-25T19:15:59.625 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:15:59.625 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:59.625 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:15:59.625 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:15:59.625 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:15:59.625 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:59.626 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.625+0000 7fd6d220d640 1 Processor -- start 2024-03-25T19:15:59.626 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.625+0000 7fd6d220d640 1 -- start start 2024-03-25T19:15:59.626 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.625+0000 7fd6d220d640 10 monclient: init 2024-03-25T19:15:59.626 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.626+0000 7fd6d220d640 10 monclient: _reopen_session rank -1 2024-03-25T19:15:59.627 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.626+0000 7fd6d220d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:15:59.627 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.626+0000 7fd6d220d640 10 monclient(hunting): picked mon.noname-a con 0x7fd6cc0b4e70 addr v1:172.21.15.6:6789/0 2024-03-25T19:15:59.627 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.626+0000 7fd6d220d640 10 monclient(hunting): picked mon.noname-c con 0x7fd6cc0b44b0 addr v1:172.21.15.192:6789/0 2024-03-25T19:15:59.627 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.626+0000 7fd6d220d640 10 monclient(hunting): picked mon.noname-b con 0x7fd6cc0b1b00 addr v1:172.21.15.6:6790/0 2024-03-25T19:15:59.627 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.626+0000 7fd6d220d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6cc0a96f0 con 0x7fd6cc0b4e70 2024-03-25T19:15:59.627 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.626+0000 7fd6d220d640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6cc0ab0e0 con 0x7fd6cc0b44b0 2024-03-25T19:15:59.627 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.626+0000 7fd6d220d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6cc145f70 con 0x7fd6cc0b1b00 2024-03-25T19:15:59.627 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.626+0000 7fd6d220d640 10 monclient(hunting): _renew_subs 2024-03-25T19:15:59.627 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.626+0000 7fd6d220d640 10 monclient(hunting): authenticate will time out at 6063.866202s 2024-03-25T19:15:59.636 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.637+0000 7fd6caffd640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fd6cc0b44b0 0x7fd6cc145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:50168/0 (socket says 172.21.15.6:50168) 2024-03-25T19:15:59.637 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.637+0000 7fd6caffd640 1 -- 172.21.15.6:0/2584527593 learned_addr learned my addr 172.21.15.6:0/2584527593 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:15:59.649 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.649+0000 7fd6ca7fc640 1 -- 172.21.15.6:0/2584527593 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2379466652 0 0) 0x7fd6cc0a96f0 con 0x7fd6cc0b4e70 2024-03-25T19:15:59.649 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.649+0000 7fd6ca7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:15:59.649 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.649+0000 7fd6ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:15:59.650 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.649+0000 7fd6ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:15:59.650 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.649+0000 7fd6ca7fc640 10 monclient(hunting): my global_id is 4302 2024-03-25T19:15:59.650 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.649+0000 7fd6ca7fc640 1 -- 172.21.15.6:0/2584527593 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6b4003520 con 0x7fd6cc0b4e70 2024-03-25T19:15:59.651 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.651+0000 7fd6ca7fc640 1 -- 172.21.15.6:0/2584527593 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4218648603 0 0) 0x7fd6cc0ab0e0 con 0x7fd6cc0b44b0 2024-03-25T19:15:59.651 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.651+0000 7fd6ca7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:15:59.651 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.651+0000 7fd6ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:15:59.652 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.651+0000 7fd6ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:15:59.652 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.651+0000 7fd6ca7fc640 10 monclient(hunting): my global_id is 4303 2024-03-25T19:15:59.652 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.651+0000 7fd6ca7fc640 1 -- 172.21.15.6:0/2584527593 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6cc0a96f0 con 0x7fd6cc0b44b0 2024-03-25T19:15:59.656 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.656+0000 7fd6ca7fc640 1 -- 172.21.15.6:0/2584527593 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2985808325 0 0) 0x7fd6cc145f70 con 0x7fd6cc0b1b00 2024-03-25T19:15:59.656 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.656+0000 7fd6ca7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:15:59.656 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.656+0000 7fd6ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:15:59.656 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.656+0000 7fd6ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:15:59.656 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.656+0000 7fd6ca7fc640 10 monclient(hunting): my global_id is 4307 2024-03-25T19:15:59.656 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.656+0000 7fd6ca7fc640 1 -- 172.21.15.6:0/2584527593 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6cc0ab0e0 con 0x7fd6cc0b1b00 2024-03-25T19:15:59.658 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.657+0000 7fd6ca7fc640 1 -- 172.21.15.6:0/2584527593 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 689605142 0 0) 0x7fd6cc0a96f0 con 0x7fd6cc0b44b0 2024-03-25T19:15:59.658 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.657+0000 7fd6ca7fc640 1 -- 172.21.15.6:0/2584527593 >> v1:172.21.15.6:6790/0 conn(0x7fd6cc0b1b00 legacy=0x7fd6cc14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:59.658 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.657+0000 7fd6ca7fc640 1 -- 172.21.15.6:0/2584527593 >> v1:172.21.15.6:6789/0 conn(0x7fd6cc0b4e70 legacy=0x7fd6cc13f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:59.658 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.657+0000 7fd6ca7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:15:59.658 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.657+0000 7fd6ca7fc640 1 monclient: found mon.noname-c 2024-03-25T19:15:59.658 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.657+0000 7fd6ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:15:59.658 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.658+0000 7fd6ca7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:15:59.658 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.658+0000 7fd6ca7fc640 1 -- 172.21.15.6:0/2584527593 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6cc0a9b80 con 0x7fd6cc0b44b0 2024-03-25T19:15:59.658 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.658+0000 7fd6ca7fc640 10 monclient: _finish_auth 0 2024-03-25T19:15:59.658 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.658+0000 7fd6ca7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:15:59.658 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.658+0000 7fd6ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:15:59.659 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.658+0000 7fd6ca7fc640 1 -- 172.21.15.6:0/2584527593 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fd6c0001670 con 0x7fd6cc0b44b0 2024-03-25T19:15:59.659 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.658+0000 7fd6d220d640 5 monclient: authenticate success, global_id 4303 2024-03-25T19:15:59.659 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.658+0000 7fd6d220d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:15:59.659 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.658+0000 7fd6ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:15:59.659 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.658+0000 7fd6ca7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:15:59.659 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.658+0000 7fd6ca7fc640 10 monclient: dump: 2024-03-25T19:15:59.659 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:15:59.659 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:59.659 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:59.659 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:59.659 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:15:59.659 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:59.659 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:15:59.659 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:15:59.659 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:15:59.659 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:59.659 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.658+0000 7fd6d220d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:15:59.666 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.665+0000 7fd6ca7fc640 1 -- 172.21.15.6:0/2584527593 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd6c00017d0 con 0x7fd6cc0b44b0 2024-03-25T19:15:59.666 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.666+0000 7fd6ca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:15:59.666 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.666+0000 7fd6ca7fc640 1 -- 172.21.15.6:0/2584527593 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fd6c0001be0 con 0x7fd6cc0b44b0 2024-03-25T19:15:59.666 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.666+0000 7fd6ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:15:59.666 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.666+0000 7fd6ca7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:15:59.666 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.666+0000 7fd6ca7fc640 10 monclient: dump: 2024-03-25T19:15:59.666 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:15:59.666 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:59.666 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:59.666 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:59.666 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:15:59.667 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:59.667 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:15:59.667 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:15:59.667 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:15:59.667 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:59.667 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.666+0000 7fd6d220d640 10 monclient: get_monmap_and_config success 2024-03-25T19:15:59.667 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.666+0000 7fd6d220d640 4 set_mon_vals no callback set 2024-03-25T19:15:59.669 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.666+0000 7fd6d220d640 10 monclient: shutdown 2024-03-25T19:15:59.669 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.666+0000 7fd6d220d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:15:59.669 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.666+0000 7fd6d220d640 1 -- 172.21.15.6:0/2584527593 >> v1:172.21.15.192:6789/0 conn(0x7fd6cc0b44b0 legacy=0x7fd6cc145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:59.669 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.666+0000 7fd6d220d640 1 -- 172.21.15.6:0/2584527593 shutdown_connections 2024-03-25T19:15:59.669 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.666+0000 7fd6d220d640 1 -- 172.21.15.6:0/2584527593 >> 172.21.15.6:0/2584527593 conn(0x7fd6cc0617a0 msgr2=0x7fd6cc061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:15:59.670 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.667+0000 7fd6d220d640 1 -- 172.21.15.6:0/2584527593 shutdown_connections 2024-03-25T19:15:59.670 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.667+0000 7fd6d220d640 1 -- 172.21.15.6:0/2584527593 wait complete. 2024-03-25T19:15:59.670 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.667+0000 7fd6d220d640 10 monclient: build_initial_monmap 2024-03-25T19:15:59.670 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.667+0000 7fd6d220d640 10 monclient: monmap: 2024-03-25T19:15:59.670 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:15:59.670 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:15:59.670 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:15:59.670 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:15:59.670 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:15:59.670 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:15:59.670 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:15:59.670 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:15:59.670 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:15:59.671 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:59.671 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.668+0000 7fd6d220d640 1 librados: starting msgr at 2024-03-25T19:15:59.671 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.668+0000 7fd6d220d640 1 librados: starting objecter 2024-03-25T19:15:59.671 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.668+0000 7fd6d220d640 1 Processor -- start 2024-03-25T19:15:59.671 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.668+0000 7fd6d220d640 1 -- start start 2024-03-25T19:15:59.671 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.668+0000 7fd6d220d640 1 librados: setting wanted keys 2024-03-25T19:15:59.671 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.668+0000 7fd6d220d640 1 librados: calling monclient init 2024-03-25T19:15:59.671 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.668+0000 7fd6d220d640 10 monclient: init 2024-03-25T19:15:59.671 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.669+0000 7fd6d220d640 10 monclient: _reopen_session rank -1 2024-03-25T19:15:59.671 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.669+0000 7fd6d220d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:15:59.671 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.669+0000 7fd6d220d640 10 monclient(hunting): picked mon.noname-a con 0x7fd6cc0b1b00 addr v1:172.21.15.6:6789/0 2024-03-25T19:15:59.671 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.669+0000 7fd6d220d640 10 monclient(hunting): picked mon.noname-b con 0x7fd6cc0b44b0 addr v1:172.21.15.192:6789/0 2024-03-25T19:15:59.671 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.669+0000 7fd6d220d640 10 monclient(hunting): picked mon.noname-c con 0x7fd6cc0b4e70 addr v1:172.21.15.6:6790/0 2024-03-25T19:15:59.671 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.669+0000 7fd6d220d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6cc1376b0 con 0x7fd6cc0b1b00 2024-03-25T19:15:59.671 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.669+0000 7fd6d220d640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6cc137990 con 0x7fd6cc0b44b0 2024-03-25T19:15:59.671 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.669+0000 7fd6d220d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6cc136780 con 0x7fd6cc0b4e70 2024-03-25T19:15:59.672 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.669+0000 7fd6d220d640 10 monclient(hunting): _renew_subs 2024-03-25T19:15:59.672 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.669+0000 7fd6d220d640 10 monclient(hunting): authenticate will time out at 6063.909732s 2024-03-25T19:15:59.681 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.681+0000 7fd6cbfff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fd6cc0b4e70 0x7fd6cc1e2020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:44436/0 (socket says 172.21.15.6:44436) 2024-03-25T19:15:59.681 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.681+0000 7fd6cbfff640 1 -- 172.21.15.6:0/733325834 learned_addr learned my addr 172.21.15.6:0/733325834 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:15:59.683 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.683+0000 7fd6b3fff640 10 client.?.objecter ms_handle_connect 0x7fd6cc0b4e70 2024-03-25T19:15:59.683 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.683+0000 7fd6b3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:15:59.685 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.685+0000 7fd6b3fff640 10 client.?.objecter ms_handle_connect 0x7fd6cc0b1b00 2024-03-25T19:15:59.685 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.685+0000 7fd6b3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:15:59.691 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.691+0000 7fd6b3fff640 10 client.?.objecter ms_handle_connect 0x7fd6cc0b44b0 2024-03-25T19:15:59.691 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.691+0000 7fd6b3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:15:59.692 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.693+0000 7fd6b3fff640 1 -- 172.21.15.6:0/733325834 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2573546414 0 0) 0x7fd6cc1376b0 con 0x7fd6cc0b1b00 2024-03-25T19:15:59.693 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.693+0000 7fd6b3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:15:59.693 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.693+0000 7fd6b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:15:59.693 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.693+0000 7fd6b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:15:59.693 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.693+0000 7fd6b3fff640 10 monclient(hunting): my global_id is 4305 2024-03-25T19:15:59.693 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.693+0000 7fd6b3fff640 1 -- 172.21.15.6:0/733325834 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6b8004eb0 con 0x7fd6cc0b1b00 2024-03-25T19:15:59.706 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.706+0000 7fd6b3fff640 1 -- 172.21.15.6:0/733325834 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4183355535 0 0) 0x7fd6cc137990 con 0x7fd6cc0b44b0 2024-03-25T19:15:59.706 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.706+0000 7fd6b3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:15:59.706 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.706+0000 7fd6b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:15:59.707 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.706+0000 7fd6b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:15:59.707 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.706+0000 7fd6b3fff640 10 monclient(hunting): my global_id is 4306 2024-03-25T19:15:59.707 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.706+0000 7fd6b3fff640 1 -- 172.21.15.6:0/733325834 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6cc1376b0 con 0x7fd6cc0b44b0 2024-03-25T19:15:59.714 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.714+0000 7fd6b3fff640 1 -- 172.21.15.6:0/733325834 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3874214868 0 0) 0x7fd6cc136780 con 0x7fd6cc0b4e70 2024-03-25T19:15:59.714 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.714+0000 7fd6b3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:15:59.714 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.714+0000 7fd6b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:15:59.715 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.714+0000 7fd6b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:15:59.715 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.714+0000 7fd6b3fff640 10 monclient(hunting): my global_id is 4310 2024-03-25T19:15:59.715 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.714+0000 7fd6b3fff640 1 -- 172.21.15.6:0/733325834 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6cc137990 con 0x7fd6cc0b4e70 2024-03-25T19:15:59.722 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.722+0000 7fd6b3fff640 1 -- 172.21.15.6:0/733325834 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3118671902 0 0) 0x7fd6b8004eb0 con 0x7fd6cc0b1b00 2024-03-25T19:15:59.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.722+0000 7fd6b3fff640 1 -- 172.21.15.6:0/733325834 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd6cc136780 con 0x7fd6cc0b1b00 2024-03-25T19:15:59.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.722+0000 7fd6b3fff640 1 -- 172.21.15.6:0/733325834 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fd6bc0017d0 con 0x7fd6cc0b1b00 2024-03-25T19:15:59.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.722+0000 7fd6b3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:15:59.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.722+0000 7fd6b3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:15:59.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.722+0000 7fd6b3fff640 10 monclient(hunting): dump: 2024-03-25T19:15:59.723 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:15:59.723 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:59.723 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:59.723 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:59.723 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:15:59.723 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:59.723 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:15:59.723 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:15:59.724 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:15:59.724 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:59.724 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.722+0000 7fd6b3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:15:59.724 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.723+0000 7fd6b3fff640 1 -- 172.21.15.6:0/733325834 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1486880418 0 0) 0x7fd6cc1376b0 con 0x7fd6cc0b44b0 2024-03-25T19:15:59.724 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.724+0000 7fd6b3fff640 1 -- 172.21.15.6:0/733325834 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd6b8004eb0 con 0x7fd6cc0b44b0 2024-03-25T19:15:59.724 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.724+0000 7fd6b3fff640 1 -- 172.21.15.6:0/733325834 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fd6c0001ae0 con 0x7fd6cc0b44b0 2024-03-25T19:15:59.725 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.724+0000 7fd6b3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:15:59.725 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.724+0000 7fd6b3fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:15:59.725 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.724+0000 7fd6b3fff640 10 monclient(hunting): dump: 2024-03-25T19:15:59.725 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:15:59.725 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:15:59.725 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:59.725 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:15:59.725 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:15:59.725 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:15:59.725 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:15:59.725 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:15:59.725 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:15:59.725 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:15:59.731 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.731+0000 7fd6b3fff640 1 -- 172.21.15.6:0/733325834 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2407983333 0 0) 0x7fd6cc136780 con 0x7fd6cc0b1b00 2024-03-25T19:15:59.731 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.731+0000 7fd6b3fff640 1 -- 172.21.15.6:0/733325834 >> v1:172.21.15.6:6790/0 conn(0x7fd6cc0b4e70 legacy=0x7fd6cc1e2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:59.731 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.731+0000 7fd6b3fff640 1 -- 172.21.15.6:0/733325834 >> v1:172.21.15.192:6789/0 conn(0x7fd6cc0b44b0 legacy=0x7fd6cc138810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:15:59.731 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.731+0000 7fd6b3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:15:59.732 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.731+0000 7fd6b3fff640 1 monclient: found mon.a 2024-03-25T19:15:59.732 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.731+0000 7fd6b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:15:59.732 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.731+0000 7fd6b3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:15:59.732 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.731+0000 7fd6b3fff640 1 -- 172.21.15.6:0/733325834 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6cc135880 con 0x7fd6cc0b1b00 2024-03-25T19:15:59.732 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.731+0000 7fd6b3fff640 10 monclient: _finish_auth 0 2024-03-25T19:15:59.732 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.731+0000 7fd6b3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:15:59.732 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.731+0000 7fd6b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:15:59.732 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.731+0000 7fd6d220d640 5 monclient: authenticate success, global_id 4305 2024-03-25T19:15:59.732 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.731+0000 7fd6d220d640 10 monclient: _renew_subs 2024-03-25T19:15:59.732 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.731+0000 7fd6d220d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:15:59.732 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.731+0000 7fd6d220d640 1 -- 172.21.15.6:0/733325834 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6cc135b30 con 0x7fd6cc0b1b00 2024-03-25T19:15:59.732 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.731+0000 7fd6d220d640 10 client.4305.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:15:59.732 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.731+0000 7fd6d220d640 10 monclient: _renew_subs 2024-03-25T19:15:59.733 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.731+0000 7fd6d220d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:15:59.733 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.731+0000 7fd6d220d640 1 -- 172.21.15.6:0/733325834 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6cc136200 con 0x7fd6cc0b1b00 2024-03-25T19:15:59.733 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.731+0000 7fd6d220d640 1 librados: init done 2024-03-25T19:15:59.733 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.732+0000 7fd6d220d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:15:59.733 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.732+0000 7fd6d220d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:15:59.733 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.732+0000 7fd6d220d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:15:59.733 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:15:59.732+0000 7fd6d220d640 1 -- 172.21.15.6:0/733325834 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6cc136910 con 0x7fd6cc0b1b00 2024-03-25T19:16:00.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:00.044+0000 7fd6b3fff640 1 -- 172.21.15.6:0/733325834 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd6bc001ce0 con 0x7fd6cc0b1b00 2024-03-25T19:16:00.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:00.044+0000 7fd6b3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:16:00.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:00.044+0000 7fd6b3fff640 1 -- 172.21.15.6:0/733325834 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fd6bc0037b0 con 0x7fd6cc0b1b00 2024-03-25T19:16:00.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:00.044+0000 7fd6b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:16:00.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:00.044+0000 7fd6b3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:16:00.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:00.044+0000 7fd6b3fff640 10 monclient: dump: 2024-03-25T19:16:00.046 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:16:00.046 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:00.046 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:00.046 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:00.046 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:16:00.046 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:00.046 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:16:00.046 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:16:00.046 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:16:00.046 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:00.046 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:00.044+0000 7fd6c9ffb640 4 set_mon_vals no callback set 2024-03-25T19:16:00.046 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:00.045+0000 7fd6b3fff640 1 -- 172.21.15.6:0/733325834 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7fd6bc0197c0 con 0x7fd6cc0b1b00 2024-03-25T19:16:00.047 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:00.046+0000 7fd6caffd640 10 monclient: get_auth_request con 0x7fd6b806df20 auth_method 0 2024-03-25T19:16:00.047 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:00.046+0000 7fd6b3fff640 10 client.4305.objecter ms_handle_connect 0x7fd6b806df20 2024-03-25T19:16:00.054 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:00.054+0000 7fd6b3fff640 1 -- 172.21.15.6:0/733325834 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7fd6bc053580 con 0x7fd6cc0b1b00 2024-03-25T19:16:00.054 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:00.054+0000 7fd6b3fff640 10 client.4305.objecter ms_dispatch 0x7fd6cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:16:00.054 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:00.054+0000 7fd6b3fff640 3 client.4305.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:16:00.054 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:00.054+0000 7fd6b3fff640 3 client.4305.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:16:00.055 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:00.054+0000 7fd6b3fff640 20 client.4305.objecter dump_active .. 0 homeless 2024-03-25T19:16:00.055 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:00.054+0000 7fd6b3fff640 1 -- 172.21.15.6:0/733325834 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd6bc04eec0 con 0x7fd6cc0b1b00 2024-03-25T19:16:00.055 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:00.054+0000 7fd6b3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:16:00.055 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:00.054+0000 7fd6b3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:16:00.301 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:00.301+0000 7fd6d220d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:16:00.301 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:00.301+0000 7fd6d220d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:16:00.301 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:00.301+0000 7fd6d220d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:16:00.301 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:00.301+0000 7fd6d220d640 1 -- 172.21.15.6:0/733325834 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd6cc04f730 con 0x7fd6cc0b1b00 2024-03-25T19:16:00.374 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:00.373+0000 7fd6b3fff640 1 -- 172.21.15.6:0/733325834 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7fd6bc050fc0 con 0x7fd6cc0b1b00 2024-03-25T19:16:00.374 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:00.374+0000 7fd6b3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:16:00.374 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:00.374+0000 7fd6b3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:16:00.374 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:16:00.374 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:16:00.377 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:00.376+0000 7fd6d220d640 10 librados: watch_flush enter 2024-03-25T19:16:00.377 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:00.376+0000 7fd6d220d640 10 librados: watch_flush exit 2024-03-25T19:16:00.377 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:00.376+0000 7fd6d220d640 20 client.4305.objecter shutdown clearing up homeless session... 2024-03-25T19:16:00.377 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:00.376+0000 7fd6d220d640 10 client.4305.objecter successfully canceled tick 2024-03-25T19:16:00.377 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:00.376+0000 7fd6d220d640 1 -- 172.21.15.6:0/733325834 >> v1:172.21.15.6:6800/2936652650 conn(0x7fd6b806df20 legacy=0x7fd6b80703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:00.377 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:00.377+0000 7fd6d220d640 10 monclient: shutdown 2024-03-25T19:16:00.377 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:00.377+0000 7fd6d220d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:16:00.377 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:00.377+0000 7fd6d220d640 1 -- 172.21.15.6:0/733325834 >> v1:172.21.15.6:6789/0 conn(0x7fd6cc0b1b00 legacy=0x7fd6cc138100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:00.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:00.377+0000 7fd6d220d640 1 -- 172.21.15.6:0/733325834 shutdown_connections 2024-03-25T19:16:00.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:00.377+0000 7fd6d220d640 1 -- 172.21.15.6:0/733325834 >> 172.21.15.6:0/733325834 conn(0x7fd6cc0617a0 msgr2=0x7fd6cc13eb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:16:00.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:00.377+0000 7fd6d220d640 1 -- 172.21.15.6:0/733325834 shutdown_connections 2024-03-25T19:16:00.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:00.377+0000 7fd6d220d640 1 -- 172.21.15.6:0/733325834 wait complete. 2024-03-25T19:16:00.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:00.377+0000 7fd6d220d640 1 librados: shutdown 2024-03-25T19:16:00.389 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:16:02.427 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:02:19.774 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:16:02.478 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:02:19.825 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:16:02.519 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:02:19.866 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:16:02.539 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:02:19.857 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:16:02.546 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:02:19.864 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:16:02.551 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:02:19.898 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:16:02.560 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:02:19.907 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:16:02.566 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:02:19.884 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:16:02.584 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:02:19.931 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:16:07.197 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:16:07.310 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.310+0000 7f912ef3b640 10 monclient: get_monmap_and_config 2024-03-25T19:16:07.310 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.310+0000 7f912ef3b640 10 monclient: build_initial_monmap 2024-03-25T19:16:07.310 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.310+0000 7f912ef3b640 10 monclient: monmap: 2024-03-25T19:16:07.310 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:16:07.310 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:07.310 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:16:07.311108+0000 2024-03-25T19:16:07.310 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:16:07.311108+0000 2024-03-25T19:16:07.310 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:16:07.310 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:07.310 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:16:07.310 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:16:07.310 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:16:07.311 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:07.312 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.310+0000 7f912ef3b640 1 Processor -- start 2024-03-25T19:16:07.312 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.310+0000 7f912ef3b640 1 -- start start 2024-03-25T19:16:07.312 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.310+0000 7f912ef3b640 10 monclient: init 2024-03-25T19:16:07.312 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.311+0000 7f912ef3b640 10 monclient: _reopen_session rank -1 2024-03-25T19:16:07.312 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.311+0000 7f912ef3b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:16:07.312 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.311+0000 7f912ef3b640 10 monclient(hunting): picked mon.noname-a con 0x7f91280b4ff0 addr v1:172.21.15.6:6789/0 2024-03-25T19:16:07.312 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.311+0000 7f912ef3b640 10 monclient(hunting): picked mon.noname-b con 0x7f91280b4670 addr v1:172.21.15.6:6790/0 2024-03-25T19:16:07.312 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.311+0000 7f912ef3b640 10 monclient(hunting): picked mon.noname-c con 0x7f912813c480 addr v1:172.21.15.192:6789/0 2024-03-25T19:16:07.312 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.311+0000 7f912ef3b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91280ab0e0 con 0x7f91280b4ff0 2024-03-25T19:16:07.312 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.312+0000 7f912ef3b640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91280ab3c0 con 0x7f912813c480 2024-03-25T19:16:07.312 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.312+0000 7f912ef3b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91280a9f30 con 0x7f91280b4670 2024-03-25T19:16:07.312 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.312+0000 7f912ef3b640 10 monclient(hunting): _renew_subs 2024-03-25T19:16:07.313 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.312+0000 7f912ef3b640 10 monclient(hunting): authenticate will time out at 6071.551800s 2024-03-25T19:16:07.326 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.326+0000 7f912d4d8640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f912813c480 0x7f912814dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:54780/0 (socket says 172.21.15.6:54780) 2024-03-25T19:16:07.326 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.326+0000 7f912d4d8640 1 -- 172.21.15.6:0/3011281522 learned_addr learned my addr 172.21.15.6:0/3011281522 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:16:07.337 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.337+0000 7f91277fe640 1 -- 172.21.15.6:0/3011281522 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3355602930 0 0) 0x7f91280ab0e0 con 0x7f91280b4ff0 2024-03-25T19:16:07.337 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.337+0000 7f91277fe640 10 monclient(hunting): _negotiate 2024-03-25T19:16:07.337 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.337+0000 7f91277fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:16:07.337 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.337+0000 7f91277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:16:07.337 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.337+0000 7f91277fe640 10 monclient(hunting): my global_id is 4308 2024-03-25T19:16:07.337 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.337+0000 7f91277fe640 1 -- 172.21.15.6:0/3011281522 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9118003520 con 0x7f91280b4ff0 2024-03-25T19:16:07.343 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.343+0000 7f91277fe640 1 -- 172.21.15.6:0/3011281522 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2452580527 0 0) 0x7f91280ab3c0 con 0x7f912813c480 2024-03-25T19:16:07.343 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.343+0000 7f91277fe640 10 monclient(hunting): _negotiate 2024-03-25T19:16:07.343 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.343+0000 7f91277fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:16:07.343 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.343+0000 7f91277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:16:07.343 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.343+0000 7f91277fe640 10 monclient(hunting): my global_id is 4309 2024-03-25T19:16:07.343 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.343+0000 7f91277fe640 1 -- 172.21.15.6:0/3011281522 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91280ab0e0 con 0x7f912813c480 2024-03-25T19:16:07.349 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.349+0000 7f91277fe640 1 -- 172.21.15.6:0/3011281522 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3152959289 0 0) 0x7f91280ab0e0 con 0x7f912813c480 2024-03-25T19:16:07.349 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.349+0000 7f91277fe640 1 -- 172.21.15.6:0/3011281522 >> v1:172.21.15.6:6790/0 conn(0x7f91280b4670 legacy=0x7f91281425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:07.349 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.349+0000 7f91277fe640 1 -- 172.21.15.6:0/3011281522 >> v1:172.21.15.6:6789/0 conn(0x7f91280b4ff0 legacy=0x7f912813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:07.349 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.349+0000 7f91277fe640 10 monclient: _finish_hunting 0 2024-03-25T19:16:07.349 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.349+0000 7f91277fe640 1 monclient: found mon.noname-c 2024-03-25T19:16:07.349 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.349+0000 7f91277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:16:07.349 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.349+0000 7f91277fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:16:07.349 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.349+0000 7f91277fe640 1 -- 172.21.15.6:0/3011281522 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91280a9b70 con 0x7f912813c480 2024-03-25T19:16:07.350 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.349+0000 7f91277fe640 10 monclient: _finish_auth 0 2024-03-25T19:16:07.350 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.349+0000 7f91277fe640 10 monclient: _check_auth_tickets 2024-03-25T19:16:07.350 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.349+0000 7f91277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:16:07.350 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.349+0000 7f91277fe640 1 -- 172.21.15.6:0/3011281522 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f91200019f0 con 0x7f912813c480 2024-03-25T19:16:07.350 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.349+0000 7f91277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:16:07.350 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.349+0000 7f91277fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:16:07.350 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.349+0000 7f91277fe640 10 monclient: dump: 2024-03-25T19:16:07.350 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:16:07.350 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:07.350 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:07.350 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:07.350 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:16:07.350 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:07.350 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:16:07.351 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:16:07.351 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:16:07.351 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:07.351 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.349+0000 7f912ef3b640 5 monclient: authenticate success, global_id 4309 2024-03-25T19:16:07.351 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.349+0000 7f912ef3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:16:07.370 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.370+0000 7f91277fe640 1 -- 172.21.15.6:0/3011281522 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9120001b50 con 0x7f912813c480 2024-03-25T19:16:07.370 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.370+0000 7f91277fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:16:07.371 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.370+0000 7f91277fe640 1 -- 172.21.15.6:0/3011281522 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f9120001fa0 con 0x7f912813c480 2024-03-25T19:16:07.371 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.370+0000 7f91277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:16:07.371 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.370+0000 7f91277fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:16:07.371 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.370+0000 7f91277fe640 10 monclient: dump: 2024-03-25T19:16:07.371 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:16:07.371 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:07.371 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:07.371 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:07.371 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:16:07.371 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:07.371 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:16:07.371 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:16:07.371 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:16:07.371 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:07.371 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.370+0000 7f912ef3b640 10 monclient: get_monmap_and_config success 2024-03-25T19:16:07.372 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.370+0000 7f912ef3b640 4 set_mon_vals no callback set 2024-03-25T19:16:07.373 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.371+0000 7f912ef3b640 10 monclient: shutdown 2024-03-25T19:16:07.373 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.371+0000 7f912ef3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:16:07.373 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.371+0000 7f912ef3b640 1 -- 172.21.15.6:0/3011281522 >> v1:172.21.15.192:6789/0 conn(0x7f912813c480 legacy=0x7f912814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:07.373 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.371+0000 7f912ef3b640 1 -- 172.21.15.6:0/3011281522 shutdown_connections 2024-03-25T19:16:07.373 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.372+0000 7f912ef3b640 1 -- 172.21.15.6:0/3011281522 >> 172.21.15.6:0/3011281522 conn(0x7f91280617a0 msgr2=0x7f9128061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:16:07.373 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.372+0000 7f912ef3b640 1 -- 172.21.15.6:0/3011281522 shutdown_connections 2024-03-25T19:16:07.373 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.372+0000 7f912ef3b640 1 -- 172.21.15.6:0/3011281522 wait complete. 2024-03-25T19:16:07.373 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.373+0000 7f912ef3b640 10 monclient: build_initial_monmap 2024-03-25T19:16:07.373 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.373+0000 7f912ef3b640 10 monclient: monmap: 2024-03-25T19:16:07.373 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:16:07.373 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:16:07.373 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:16:07.373 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:16:07.373 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:16:07.374 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:16:07.374 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:16:07.374 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:16:07.374 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:16:07.374 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:07.375 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.373+0000 7f912ef3b640 1 librados: starting msgr at 2024-03-25T19:16:07.375 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.373+0000 7f912ef3b640 1 librados: starting objecter 2024-03-25T19:16:07.375 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.373+0000 7f912ef3b640 1 Processor -- start 2024-03-25T19:16:07.375 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.374+0000 7f912ef3b640 1 -- start start 2024-03-25T19:16:07.375 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.374+0000 7f912ef3b640 1 librados: setting wanted keys 2024-03-25T19:16:07.376 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.374+0000 7f912ef3b640 1 librados: calling monclient init 2024-03-25T19:16:07.376 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.374+0000 7f912ef3b640 10 monclient: init 2024-03-25T19:16:07.376 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.375+0000 7f912ef3b640 10 monclient: _reopen_session rank -1 2024-03-25T19:16:07.376 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.375+0000 7f912ef3b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:16:07.376 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.375+0000 7f912ef3b640 10 monclient(hunting): picked mon.noname-b con 0x7f91280b4670 addr v1:172.21.15.192:6789/0 2024-03-25T19:16:07.376 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.375+0000 7f912ef3b640 10 monclient(hunting): picked mon.noname-a con 0x7f91280b4ff0 addr v1:172.21.15.6:6789/0 2024-03-25T19:16:07.376 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.375+0000 7f912ef3b640 10 monclient(hunting): picked mon.noname-c con 0x7f912813c480 addr v1:172.21.15.6:6790/0 2024-03-25T19:16:07.376 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.375+0000 7f912ef3b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9128144720 con 0x7f91280b4ff0 2024-03-25T19:16:07.376 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.375+0000 7f912ef3b640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f912813eb10 con 0x7f91280b4670 2024-03-25T19:16:07.376 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.375+0000 7f912ef3b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f912813f1a0 con 0x7f912813c480 2024-03-25T19:16:07.376 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.375+0000 7f912ef3b640 10 monclient(hunting): _renew_subs 2024-03-25T19:16:07.376 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.375+0000 7f912ef3b640 10 monclient(hunting): authenticate will time out at 6071.615608s 2024-03-25T19:16:07.391 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.391+0000 7f912d4d8640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f912813c480 0x7f912813d020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:49098/0 (socket says 172.21.15.6:49098) 2024-03-25T19:16:07.391 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.391+0000 7f912d4d8640 1 -- 172.21.15.6:0/2302715640 learned_addr learned my addr 172.21.15.6:0/2302715640 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:16:07.392 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.392+0000 7f91257fa640 10 client.?.objecter ms_handle_connect 0x7f912813c480 2024-03-25T19:16:07.392 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.392+0000 7f91257fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:16:07.394 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.394+0000 7f91257fa640 10 client.?.objecter ms_handle_connect 0x7f91280b4670 2024-03-25T19:16:07.394 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.394+0000 7f91257fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:16:07.402 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.402+0000 7f91257fa640 10 client.?.objecter ms_handle_connect 0x7f91280b4ff0 2024-03-25T19:16:07.402 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.402+0000 7f91257fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:16:07.414 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.414+0000 7f91257fa640 1 -- 172.21.15.6:0/2302715640 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3264412336 0 0) 0x7f912813f1a0 con 0x7f912813c480 2024-03-25T19:16:07.414 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.414+0000 7f91257fa640 10 monclient(hunting): _negotiate 2024-03-25T19:16:07.414 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.414+0000 7f91257fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:16:07.414 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.414+0000 7f91257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:16:07.414 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.414+0000 7f91257fa640 10 monclient(hunting): my global_id is 4316 2024-03-25T19:16:07.414 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.414+0000 7f91257fa640 1 -- 172.21.15.6:0/2302715640 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f910c004eb0 con 0x7f912813c480 2024-03-25T19:16:07.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.416+0000 7f91257fa640 1 -- 172.21.15.6:0/2302715640 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 454120056 0 0) 0x7f912813eb10 con 0x7f91280b4670 2024-03-25T19:16:07.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.416+0000 7f91257fa640 10 monclient(hunting): _negotiate 2024-03-25T19:16:07.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.416+0000 7f91257fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:16:07.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.416+0000 7f91257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:16:07.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.416+0000 7f91257fa640 10 monclient(hunting): my global_id is 4312 2024-03-25T19:16:07.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.416+0000 7f91257fa640 1 -- 172.21.15.6:0/2302715640 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f912813f1a0 con 0x7f91280b4670 2024-03-25T19:16:07.429 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.428+0000 7f91257fa640 1 -- 172.21.15.6:0/2302715640 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 538827874 0 0) 0x7f9128144720 con 0x7f91280b4ff0 2024-03-25T19:16:07.429 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.428+0000 7f91257fa640 10 monclient(hunting): _negotiate 2024-03-25T19:16:07.429 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.428+0000 7f91257fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:16:07.429 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.428+0000 7f91257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:16:07.429 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.428+0000 7f91257fa640 10 monclient(hunting): my global_id is 4311 2024-03-25T19:16:07.429 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.428+0000 7f91257fa640 1 -- 172.21.15.6:0/2302715640 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f912813eb10 con 0x7f91280b4ff0 2024-03-25T19:16:07.429 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.428+0000 7f91257fa640 1 -- 172.21.15.6:0/2302715640 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2294647888 0 0) 0x7f910c004eb0 con 0x7f912813c480 2024-03-25T19:16:07.429 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.428+0000 7f91257fa640 1 -- 172.21.15.6:0/2302715640 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9128144720 con 0x7f912813c480 2024-03-25T19:16:07.429 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.428+0000 7f91257fa640 1 -- 172.21.15.6:0/2302715640 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f9120001f20 con 0x7f912813c480 2024-03-25T19:16:07.429 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.428+0000 7f91257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:16:07.429 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.428+0000 7f91257fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:16:07.429 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.428+0000 7f91257fa640 10 monclient(hunting): dump: 2024-03-25T19:16:07.429 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:16:07.430 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:07.430 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:07.430 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:07.430 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:16:07.430 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:07.430 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:16:07.430 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:16:07.430 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:16:07.430 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:07.430 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.428+0000 7f91257fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:16:07.437 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.437+0000 7f91257fa640 1 -- 172.21.15.6:0/2302715640 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4020141745 0 0) 0x7f912813f1a0 con 0x7f91280b4670 2024-03-25T19:16:07.437 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.437+0000 7f91257fa640 1 -- 172.21.15.6:0/2302715640 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f910c004eb0 con 0x7f91280b4670 2024-03-25T19:16:07.437 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.437+0000 7f91257fa640 1 -- 172.21.15.6:0/2302715640 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f911c0018d0 con 0x7f91280b4670 2024-03-25T19:16:07.437 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.437+0000 7f91257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:16:07.437 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.437+0000 7f91257fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:16:07.437 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.437+0000 7f91257fa640 10 monclient(hunting): dump: 2024-03-25T19:16:07.438 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:16:07.438 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:07.438 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:07.438 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:07.438 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:16:07.438 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:07.438 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:16:07.438 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:16:07.438 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:16:07.438 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:07.438 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.437+0000 7f91257fa640 1 -- 172.21.15.6:0/2302715640 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3898570344 0 0) 0x7f912813eb10 con 0x7f91280b4ff0 2024-03-25T19:16:07.438 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.437+0000 7f91257fa640 1 -- 172.21.15.6:0/2302715640 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f912813f1a0 con 0x7f91280b4ff0 2024-03-25T19:16:07.438 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.437+0000 7f91257fa640 1 -- 172.21.15.6:0/2302715640 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f9114001860 con 0x7f91280b4ff0 2024-03-25T19:16:07.438 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.437+0000 7f91257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:16:07.438 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.437+0000 7f91257fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:16:07.439 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.437+0000 7f91257fa640 10 monclient(hunting): dump: 2024-03-25T19:16:07.439 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:16:07.439 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:07.439 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:07.439 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:07.439 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:16:07.439 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:07.439 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:16:07.439 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:16:07.439 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:16:07.439 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:07.442 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.442+0000 7f91257fa640 1 -- 172.21.15.6:0/2302715640 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3679894907 0 0) 0x7f9128144720 con 0x7f912813c480 2024-03-25T19:16:07.442 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.442+0000 7f91257fa640 1 -- 172.21.15.6:0/2302715640 >> v1:172.21.15.192:6789/0 conn(0x7f91280b4670 legacy=0x7f9128145190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:07.442 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.442+0000 7f91257fa640 1 -- 172.21.15.6:0/2302715640 >> v1:172.21.15.6:6789/0 conn(0x7f91280b4ff0 legacy=0x7f912813c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:07.442 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.442+0000 7f91257fa640 10 monclient: _finish_hunting 0 2024-03-25T19:16:07.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.442+0000 7f91257fa640 1 monclient: found mon.c 2024-03-25T19:16:07.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.442+0000 7f91257fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:16:07.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.442+0000 7f91257fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:16:07.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.442+0000 7f91257fa640 1 -- 172.21.15.6:0/2302715640 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9128146650 con 0x7f912813c480 2024-03-25T19:16:07.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.442+0000 7f91257fa640 10 monclient: _finish_auth 0 2024-03-25T19:16:07.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.442+0000 7f91257fa640 10 monclient: _check_auth_tickets 2024-03-25T19:16:07.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.442+0000 7f91257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:16:07.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.442+0000 7f912ef3b640 5 monclient: authenticate success, global_id 4316 2024-03-25T19:16:07.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.442+0000 7f912ef3b640 10 monclient: _renew_subs 2024-03-25T19:16:07.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.442+0000 7f912ef3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:16:07.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.442+0000 7f912ef3b640 1 -- 172.21.15.6:0/2302715640 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f912813d730 con 0x7f912813c480 2024-03-25T19:16:07.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.442+0000 7f912ef3b640 10 client.4316.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:16:07.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.442+0000 7f912ef3b640 10 monclient: _renew_subs 2024-03-25T19:16:07.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.442+0000 7f912ef3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:16:07.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.442+0000 7f912ef3b640 1 -- 172.21.15.6:0/2302715640 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f91281437a0 con 0x7f912813c480 2024-03-25T19:16:07.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.442+0000 7f912ef3b640 1 librados: init done 2024-03-25T19:16:07.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.443+0000 7f912ef3b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:16:07.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.443+0000 7f912ef3b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:16:07.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.443+0000 7f912ef3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:16:07.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.443+0000 7f912ef3b640 1 -- 172.21.15.6:0/2302715640 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9128143410 con 0x7f912813c480 2024-03-25T19:16:07.774 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.774+0000 7f91257fa640 1 -- 172.21.15.6:0/2302715640 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9120004e10 con 0x7f912813c480 2024-03-25T19:16:07.774 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.774+0000 7f91257fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:16:07.774 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.774+0000 7f91257fa640 1 -- 172.21.15.6:0/2302715640 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f9120005120 con 0x7f912813c480 2024-03-25T19:16:07.774 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.774+0000 7f91257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:16:07.774 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.774+0000 7f91257fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:16:07.774 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.774+0000 7f91257fa640 10 monclient: dump: 2024-03-25T19:16:07.775 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:16:07.775 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:07.775 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:07.775 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:07.775 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:16:07.775 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:07.775 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:16:07.775 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:16:07.775 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:16:07.775 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:07.775 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.774+0000 7f91267fc640 4 set_mon_vals no callback set 2024-03-25T19:16:07.776 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.775+0000 7f91257fa640 1 -- 172.21.15.6:0/2302715640 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f912001b0f0 con 0x7f912813c480 2024-03-25T19:16:07.776 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.775+0000 7f912ccd7640 10 monclient: get_auth_request con 0x7f910c06e0f0 auth_method 0 2024-03-25T19:16:07.776 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.775+0000 7f91257fa640 10 client.4316.objecter ms_handle_connect 0x7f910c06e0f0 2024-03-25T19:16:07.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.782+0000 7f91257fa640 1 -- 172.21.15.6:0/2302715640 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f9120054b80 con 0x7f912813c480 2024-03-25T19:16:07.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.782+0000 7f91257fa640 10 client.4316.objecter ms_dispatch 0x7f9128000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:16:07.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.782+0000 7f91257fa640 3 client.4316.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:16:07.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.782+0000 7f91257fa640 3 client.4316.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:16:07.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.782+0000 7f91257fa640 20 client.4316.objecter dump_active .. 0 homeless 2024-03-25T19:16:07.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.782+0000 7f91257fa640 1 -- 172.21.15.6:0/2302715640 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9120050690 con 0x7f912813c480 2024-03-25T19:16:07.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.782+0000 7f91257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:16:07.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:07.782+0000 7f91257fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:16:08.030 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:08.030+0000 7f912ef3b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:16:08.031 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:08.030+0000 7f912ef3b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:16:08.031 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:08.030+0000 7f912ef3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:16:08.031 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:08.030+0000 7f912ef3b640 1 -- 172.21.15.6:0/2302715640 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f91281467e0 con 0x7f912813c480 2024-03-25T19:16:08.090 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:08.090+0000 7f91257fa640 1 -- 172.21.15.6:0/2302715640 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f91200525c0 con 0x7f912813c480 2024-03-25T19:16:08.090 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:08.090+0000 7f91257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:16:08.090 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:08.090+0000 7f91257fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:16:08.090 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:16:08.090 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:16:08.093 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:08.092+0000 7f912ef3b640 10 librados: watch_flush enter 2024-03-25T19:16:08.093 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:08.092+0000 7f912ef3b640 10 librados: watch_flush exit 2024-03-25T19:16:08.093 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:08.092+0000 7f912ef3b640 20 client.4316.objecter shutdown clearing up homeless session... 2024-03-25T19:16:08.093 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:08.092+0000 7f912ef3b640 10 client.4316.objecter successfully canceled tick 2024-03-25T19:16:08.093 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:08.092+0000 7f912ef3b640 1 -- 172.21.15.6:0/2302715640 >> v1:172.21.15.6:6800/2936652650 conn(0x7f910c06e0f0 legacy=0x7f910c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:08.093 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:08.092+0000 7f912ef3b640 10 monclient: shutdown 2024-03-25T19:16:08.093 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:08.092+0000 7f912ef3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:16:08.093 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:08.092+0000 7f912ef3b640 1 -- 172.21.15.6:0/2302715640 >> v1:172.21.15.6:6790/0 conn(0x7f912813c480 legacy=0x7f912813d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:08.093 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:08.092+0000 7f912ef3b640 1 -- 172.21.15.6:0/2302715640 shutdown_connections 2024-03-25T19:16:08.093 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:08.092+0000 7f912ef3b640 1 -- 172.21.15.6:0/2302715640 >> 172.21.15.6:0/2302715640 conn(0x7f91280617a0 msgr2=0x7f91281405c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:16:08.093 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:08.093+0000 7f912ef3b640 1 -- 172.21.15.6:0/2302715640 shutdown_connections 2024-03-25T19:16:08.093 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:08.093+0000 7f912ef3b640 1 -- 172.21.15.6:0/2302715640 wait complete. 2024-03-25T19:16:08.094 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:08.093+0000 7f912ef3b640 1 librados: shutdown 2024-03-25T19:16:08.104 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:16:14.910 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:16:15.027 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.026+0000 7f2b5d64a640 10 monclient: get_monmap_and_config 2024-03-25T19:16:15.027 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.026+0000 7f2b5d64a640 10 monclient: build_initial_monmap 2024-03-25T19:16:15.027 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.027+0000 7f2b5d64a640 10 monclient: monmap: 2024-03-25T19:16:15.027 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:16:15.027 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:15.027 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:16:15.028005+0000 2024-03-25T19:16:15.027 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:16:15.028005+0000 2024-03-25T19:16:15.027 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:16:15.027 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:15.027 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:16:15.027 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:16:15.027 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:16:15.027 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:15.029 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.027+0000 7f2b5d64a640 1 Processor -- start 2024-03-25T19:16:15.029 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.027+0000 7f2b5d64a640 1 -- start start 2024-03-25T19:16:15.029 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.027+0000 7f2b5d64a640 10 monclient: init 2024-03-25T19:16:15.029 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.028+0000 7f2b5d64a640 10 monclient: _reopen_session rank -1 2024-03-25T19:16:15.029 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.028+0000 7f2b5d64a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:16:15.029 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.028+0000 7f2b5d64a640 10 monclient(hunting): picked mon.noname-a con 0x7f2b580b4ff0 addr v1:172.21.15.6:6789/0 2024-03-25T19:16:15.029 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.028+0000 7f2b5d64a640 10 monclient(hunting): picked mon.noname-c con 0x7f2b580b4670 addr v1:172.21.15.192:6789/0 2024-03-25T19:16:15.029 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.028+0000 7f2b5d64a640 10 monclient(hunting): picked mon.noname-b con 0x7f2b5813c480 addr v1:172.21.15.6:6790/0 2024-03-25T19:16:15.029 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.028+0000 7f2b5d64a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b580ab0e0 con 0x7f2b580b4ff0 2024-03-25T19:16:15.029 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.028+0000 7f2b5d64a640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b580ab3c0 con 0x7f2b580b4670 2024-03-25T19:16:15.029 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.029+0000 7f2b5d64a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b580a9f30 con 0x7f2b5813c480 2024-03-25T19:16:15.030 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.029+0000 7f2b5d64a640 10 monclient(hunting): _renew_subs 2024-03-25T19:16:15.030 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.029+0000 7f2b5d64a640 10 monclient(hunting): authenticate will time out at 6079.268994s 2024-03-25T19:16:15.032 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.032+0000 7f2b577fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f2b5813c480 0x7f2b5814dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:54498/0 (socket says 172.21.15.6:54498) 2024-03-25T19:16:15.033 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.032+0000 7f2b577fe640 1 -- 172.21.15.6:0/3161954036 learned_addr learned my addr 172.21.15.6:0/3161954036 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:16:15.038 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.038+0000 7f2b55ffb640 1 -- 172.21.15.6:0/3161954036 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2720374287 0 0) 0x7f2b580a9f30 con 0x7f2b5813c480 2024-03-25T19:16:15.038 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.038+0000 7f2b55ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:16:15.038 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.038+0000 7f2b55ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:16:15.038 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.038+0000 7f2b55ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:16:15.038 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.038+0000 7f2b55ffb640 10 monclient(hunting): my global_id is 4319 2024-03-25T19:16:15.038 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.038+0000 7f2b55ffb640 1 -- 172.21.15.6:0/3161954036 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b40003520 con 0x7f2b5813c480 2024-03-25T19:16:15.044 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.044+0000 7f2b55ffb640 1 -- 172.21.15.6:0/3161954036 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 139842739 0 0) 0x7f2b40003520 con 0x7f2b5813c480 2024-03-25T19:16:15.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.044+0000 7f2b55ffb640 1 -- 172.21.15.6:0/3161954036 >> v1:172.21.15.192:6789/0 conn(0x7f2b580b4670 legacy=0x7f2b581425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:15.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.044+0000 7f2b55ffb640 1 -- 172.21.15.6:0/3161954036 >> v1:172.21.15.6:6789/0 conn(0x7f2b580b4ff0 legacy=0x7f2b5813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:15.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.044+0000 7f2b55ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:16:15.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.044+0000 7f2b55ffb640 1 monclient: found mon.noname-b 2024-03-25T19:16:15.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.044+0000 7f2b55ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:16:15.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.044+0000 7f2b55ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-03-25T19:16:15.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.044+0000 7f2b55ffb640 1 -- 172.21.15.6:0/3161954036 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b580a9b70 con 0x7f2b5813c480 2024-03-25T19:16:15.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.044+0000 7f2b55ffb640 10 monclient: _finish_auth 0 2024-03-25T19:16:15.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.044+0000 7f2b55ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:16:15.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.044+0000 7f2b55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:16:15.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.044+0000 7f2b55ffb640 1 -- 172.21.15.6:0/3161954036 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f2b50001a10 con 0x7f2b5813c480 2024-03-25T19:16:15.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.044+0000 7f2b55ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:16:15.046 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.045+0000 7f2b55ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:16:15.046 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.045+0000 7f2b55ffb640 10 monclient: dump: 2024-03-25T19:16:15.046 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:16:15.046 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:15.046 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:15.046 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:15.046 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:16:15.046 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:15.046 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:16:15.046 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:16:15.046 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:16:15.046 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:15.046 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.045+0000 7f2b5d64a640 5 monclient: authenticate success, global_id 4319 2024-03-25T19:16:15.047 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.045+0000 7f2b5d64a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:16:15.048 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.047+0000 7f2b55ffb640 1 -- 172.21.15.6:0/3161954036 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b50001b70 con 0x7f2b5813c480 2024-03-25T19:16:15.048 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.047+0000 7f2b55ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:16:15.048 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.047+0000 7f2b5d64a640 10 monclient: get_monmap_and_config success 2024-03-25T19:16:15.048 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.047+0000 7f2b5d64a640 4 set_mon_vals no callback set 2024-03-25T19:16:15.048 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.047+0000 7f2b5d64a640 10 monclient: shutdown 2024-03-25T19:16:15.048 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.047+0000 7f2b5d64a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:16:15.048 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.047+0000 7f2b5d64a640 1 -- 172.21.15.6:0/3161954036 >> v1:172.21.15.6:6790/0 conn(0x7f2b5813c480 legacy=0x7f2b5814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:15.048 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.047+0000 7f2b5d64a640 1 -- 172.21.15.6:0/3161954036 shutdown_connections 2024-03-25T19:16:15.048 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.047+0000 7f2b5d64a640 1 -- 172.21.15.6:0/3161954036 >> 172.21.15.6:0/3161954036 conn(0x7f2b580617a0 msgr2=0x7f2b58061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:16:15.048 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.047+0000 7f2b5d64a640 1 -- 172.21.15.6:0/3161954036 shutdown_connections 2024-03-25T19:16:15.048 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.047+0000 7f2b5d64a640 1 -- 172.21.15.6:0/3161954036 wait complete. 2024-03-25T19:16:15.050 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.048+0000 7f2b5d64a640 10 monclient: build_initial_monmap 2024-03-25T19:16:15.050 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.048+0000 7f2b5d64a640 10 monclient: monmap: 2024-03-25T19:16:15.050 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:16:15.050 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:16:15.050 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:16:15.051 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:16:15.051 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:16:15.051 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:16:15.051 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:16:15.051 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:16:15.051 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:16:15.051 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:15.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.048+0000 7f2b5d64a640 1 librados: starting msgr at 2024-03-25T19:16:15.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.048+0000 7f2b5d64a640 1 librados: starting objecter 2024-03-25T19:16:15.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.049+0000 7f2b5d64a640 1 Processor -- start 2024-03-25T19:16:15.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.049+0000 7f2b5d64a640 1 -- start start 2024-03-25T19:16:15.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.049+0000 7f2b5d64a640 1 librados: setting wanted keys 2024-03-25T19:16:15.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.049+0000 7f2b5d64a640 1 librados: calling monclient init 2024-03-25T19:16:15.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.049+0000 7f2b5d64a640 10 monclient: init 2024-03-25T19:16:15.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.049+0000 7f2b5d64a640 10 monclient: _reopen_session rank -1 2024-03-25T19:16:15.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.050+0000 7f2b5d64a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:16:15.052 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.050+0000 7f2b5d64a640 10 monclient(hunting): picked mon.noname-a con 0x7f2b580b4670 addr v1:172.21.15.6:6789/0 2024-03-25T19:16:15.052 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.050+0000 7f2b5d64a640 10 monclient(hunting): picked mon.noname-c con 0x7f2b580b4ff0 addr v1:172.21.15.6:6790/0 2024-03-25T19:16:15.052 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.050+0000 7f2b5d64a640 10 monclient(hunting): picked mon.noname-b con 0x7f2b5813c480 addr v1:172.21.15.192:6789/0 2024-03-25T19:16:15.052 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.050+0000 7f2b5d64a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b5813b520 con 0x7f2b580b4670 2024-03-25T19:16:15.052 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.050+0000 7f2b5d64a640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b58071bb0 con 0x7f2b5813c480 2024-03-25T19:16:15.052 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.050+0000 7f2b5d64a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b580abc60 con 0x7f2b580b4ff0 2024-03-25T19:16:15.052 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.050+0000 7f2b5d64a640 10 monclient(hunting): _renew_subs 2024-03-25T19:16:15.052 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.050+0000 7f2b5d64a640 10 monclient(hunting): authenticate will time out at 6079.290014s 2024-03-25T19:16:15.060 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.060+0000 7f2b56ffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2b580b4670 0x7f2b5806dec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44824/0 (socket says 172.21.15.6:44824) 2024-03-25T19:16:15.060 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.060+0000 7f2b56ffd640 1 -- 172.21.15.6:0/3348337575 learned_addr learned my addr 172.21.15.6:0/3348337575 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:16:15.061 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.061+0000 7f2b3f7fe640 10 client.?.objecter ms_handle_connect 0x7f2b580b4670 2024-03-25T19:16:15.061 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.061+0000 7f2b3f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:16:15.063 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.063+0000 7f2b3f7fe640 10 client.?.objecter ms_handle_connect 0x7f2b580b4ff0 2024-03-25T19:16:15.063 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.063+0000 7f2b3f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:16:15.072 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.072+0000 7f2b3f7fe640 10 client.?.objecter ms_handle_connect 0x7f2b5813c480 2024-03-25T19:16:15.072 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.072+0000 7f2b3f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:16:15.073 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.074+0000 7f2b3f7fe640 1 -- 172.21.15.6:0/3348337575 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1481711314 0 0) 0x7f2b5813b520 con 0x7f2b580b4670 2024-03-25T19:16:15.074 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.074+0000 7f2b3f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:16:15.074 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.074+0000 7f2b3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:16:15.074 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.074+0000 7f2b3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:16:15.074 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.074+0000 7f2b3f7fe640 10 monclient(hunting): my global_id is 4317 2024-03-25T19:16:15.074 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.074+0000 7f2b3f7fe640 1 -- 172.21.15.6:0/3348337575 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b44004eb0 con 0x7f2b580b4670 2024-03-25T19:16:15.075 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.075+0000 7f2b3f7fe640 1 -- 172.21.15.6:0/3348337575 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3907861808 0 0) 0x7f2b580abc60 con 0x7f2b580b4ff0 2024-03-25T19:16:15.075 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.075+0000 7f2b3f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:16:15.075 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.075+0000 7f2b3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:16:15.075 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.075+0000 7f2b3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:16:15.076 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.075+0000 7f2b3f7fe640 10 monclient(hunting): my global_id is 4322 2024-03-25T19:16:15.076 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.075+0000 7f2b3f7fe640 1 -- 172.21.15.6:0/3348337575 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b5813b520 con 0x7f2b580b4ff0 2024-03-25T19:16:15.088 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.088+0000 7f2b3f7fe640 1 -- 172.21.15.6:0/3348337575 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3615838178 0 0) 0x7f2b58071bb0 con 0x7f2b5813c480 2024-03-25T19:16:15.088 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.088+0000 7f2b3f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:16:15.088 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.088+0000 7f2b3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:16:15.089 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.088+0000 7f2b3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:16:15.089 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.088+0000 7f2b3f7fe640 10 monclient(hunting): my global_id is 4318 2024-03-25T19:16:15.089 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.088+0000 7f2b3f7fe640 1 -- 172.21.15.6:0/3348337575 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b580abc60 con 0x7f2b5813c480 2024-03-25T19:16:15.097 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.097+0000 7f2b3f7fe640 1 -- 172.21.15.6:0/3348337575 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 642946776 0 0) 0x7f2b5813b520 con 0x7f2b580b4ff0 2024-03-25T19:16:15.098 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.097+0000 7f2b3f7fe640 1 -- 172.21.15.6:0/3348337575 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b58071bb0 con 0x7f2b580b4ff0 2024-03-25T19:16:15.098 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.097+0000 7f2b3f7fe640 1 -- 172.21.15.6:0/3348337575 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f2b4c0018b0 con 0x7f2b580b4ff0 2024-03-25T19:16:15.098 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.097+0000 7f2b3f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:16:15.098 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.097+0000 7f2b3f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:16:15.098 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.097+0000 7f2b3f7fe640 10 monclient(hunting): dump: 2024-03-25T19:16:15.098 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:16:15.098 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:15.098 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:15.098 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:15.098 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:16:15.098 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:15.098 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:16:15.098 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:16:15.099 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:16:15.099 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:15.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.097+0000 7f2b3f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:16:15.101 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.101+0000 7f2b3f7fe640 1 -- 172.21.15.6:0/3348337575 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 621582552 0 0) 0x7f2b44004eb0 con 0x7f2b580b4670 2024-03-25T19:16:15.101 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.101+0000 7f2b3f7fe640 1 -- 172.21.15.6:0/3348337575 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b5813b520 con 0x7f2b580b4670 2024-03-25T19:16:15.101 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.101+0000 7f2b3f7fe640 1 -- 172.21.15.6:0/3348337575 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f2b48001a30 con 0x7f2b580b4670 2024-03-25T19:16:15.101 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.101+0000 7f2b3f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:16:15.101 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.101+0000 7f2b3f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:16:15.101 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.101+0000 7f2b3f7fe640 10 monclient(hunting): dump: 2024-03-25T19:16:15.101 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:16:15.101 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:15.101 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:15.101 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:15.101 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:16:15.101 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:15.101 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:16:15.102 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:16:15.102 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:16:15.102 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:15.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.103+0000 7f2b3f7fe640 1 -- 172.21.15.6:0/3348337575 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1374282560 0 0) 0x7f2b58071bb0 con 0x7f2b580b4ff0 2024-03-25T19:16:15.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.103+0000 7f2b3f7fe640 1 -- 172.21.15.6:0/3348337575 >> v1:172.21.15.192:6789/0 conn(0x7f2b5813c480 legacy=0x7f2b5813ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:15.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.103+0000 7f2b3f7fe640 1 -- 172.21.15.6:0/3348337575 >> v1:172.21.15.6:6789/0 conn(0x7f2b580b4670 legacy=0x7f2b5806dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:15.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.103+0000 7f2b3f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:16:15.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.103+0000 7f2b3f7fe640 1 monclient: found mon.c 2024-03-25T19:16:15.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.103+0000 7f2b3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:16:15.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.103+0000 7f2b3f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:16:15.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.103+0000 7f2b3f7fe640 1 -- 172.21.15.6:0/3348337575 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b580abf40 con 0x7f2b580b4ff0 2024-03-25T19:16:15.105 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.103+0000 7f2b3f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:16:15.105 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.103+0000 7f2b3f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:16:15.105 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.103+0000 7f2b3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:16:15.105 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.104+0000 7f2b5d64a640 5 monclient: authenticate success, global_id 4322 2024-03-25T19:16:15.105 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.104+0000 7f2b5d64a640 10 monclient: _renew_subs 2024-03-25T19:16:15.105 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.104+0000 7f2b5d64a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:16:15.105 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.104+0000 7f2b5d64a640 1 -- 172.21.15.6:0/3348337575 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b58071690 con 0x7f2b580b4ff0 2024-03-25T19:16:15.105 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.104+0000 7f2b5d64a640 10 client.4322.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:16:15.105 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.104+0000 7f2b5d64a640 10 monclient: _renew_subs 2024-03-25T19:16:15.105 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.104+0000 7f2b5d64a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:16:15.105 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.104+0000 7f2b5d64a640 1 -- 172.21.15.6:0/3348337575 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b58070a20 con 0x7f2b580b4ff0 2024-03-25T19:16:15.105 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.104+0000 7f2b5d64a640 1 librados: init done 2024-03-25T19:16:15.106 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.105+0000 7f2b5d64a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:16:15.106 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.105+0000 7f2b5d64a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:16:15.106 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.105+0000 7f2b5d64a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:16:15.106 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.105+0000 7f2b5d64a640 1 -- 172.21.15.6:0/3348337575 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b5806dbd0 con 0x7f2b580b4ff0 2024-03-25T19:16:15.407 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.407+0000 7f2b3f7fe640 1 -- 172.21.15.6:0/3348337575 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b4c003610 con 0x7f2b580b4ff0 2024-03-25T19:16:15.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.407+0000 7f2b3f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:16:15.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.407+0000 7f2b3f7fe640 1 -- 172.21.15.6:0/3348337575 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f2b4c003a60 con 0x7f2b580b4ff0 2024-03-25T19:16:15.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.407+0000 7f2b3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:16:15.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.407+0000 7f2b3f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:16:15.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.407+0000 7f2b3f7fe640 10 monclient: dump: 2024-03-25T19:16:15.408 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:16:15.408 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:15.408 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:15.408 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:15.409 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:16:15.409 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:15.409 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:16:15.409 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:16:15.409 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:16:15.409 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:15.409 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.407+0000 7f2b557fa640 4 set_mon_vals no callback set 2024-03-25T19:16:15.410 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.408+0000 7f2b3f7fe640 1 -- 172.21.15.6:0/3348337575 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f2b4c003c40 con 0x7f2b580b4ff0 2024-03-25T19:16:15.410 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.409+0000 7f2b56ffd640 10 monclient: get_auth_request con 0x7f2b4406df20 auth_method 0 2024-03-25T19:16:15.410 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.409+0000 7f2b3f7fe640 10 client.4322.objecter ms_handle_connect 0x7f2b4406df20 2024-03-25T19:16:15.413 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.413+0000 7f2b3f7fe640 1 -- 172.21.15.6:0/3348337575 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f2b4c053620 con 0x7f2b580b4ff0 2024-03-25T19:16:15.413 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.413+0000 7f2b3f7fe640 10 client.4322.objecter ms_dispatch 0x7f2b58000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:16:15.413 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.413+0000 7f2b3f7fe640 3 client.4322.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:16:15.413 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.413+0000 7f2b3f7fe640 3 client.4322.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:16:15.414 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.413+0000 7f2b3f7fe640 20 client.4322.objecter dump_active .. 0 homeless 2024-03-25T19:16:15.414 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.413+0000 7f2b3f7fe640 1 -- 172.21.15.6:0/3348337575 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2b4c04f130 con 0x7f2b580b4ff0 2024-03-25T19:16:15.414 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.413+0000 7f2b3f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:16:15.414 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.413+0000 7f2b3f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:16:15.676 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.676+0000 7f2b5d64a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:16:15.676 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.676+0000 7f2b5d64a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:16:15.676 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.676+0000 7f2b5d64a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:16:15.676 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.676+0000 7f2b5d64a640 1 -- 172.21.15.6:0/3348337575 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2b58071820 con 0x7f2b580b4ff0 2024-03-25T19:16:15.711 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.711+0000 7f2b3f7fe640 1 -- 172.21.15.6:0/3348337575 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f2b4c051060 con 0x7f2b580b4ff0 2024-03-25T19:16:15.711 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.711+0000 7f2b3f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:16:15.711 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.711+0000 7f2b3f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:16:15.711 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:16:15.711 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:16:15.713 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.713+0000 7f2b5d64a640 10 librados: watch_flush enter 2024-03-25T19:16:15.713 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.713+0000 7f2b5d64a640 10 librados: watch_flush exit 2024-03-25T19:16:15.713 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.713+0000 7f2b5d64a640 20 client.4322.objecter shutdown clearing up homeless session... 2024-03-25T19:16:15.713 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.713+0000 7f2b5d64a640 10 client.4322.objecter successfully canceled tick 2024-03-25T19:16:15.713 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.713+0000 7f2b5d64a640 1 -- 172.21.15.6:0/3348337575 >> v1:172.21.15.6:6800/2936652650 conn(0x7f2b4406df20 legacy=0x7f2b440703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:15.714 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.713+0000 7f2b5d64a640 10 monclient: shutdown 2024-03-25T19:16:15.714 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.713+0000 7f2b5d64a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:16:15.714 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.713+0000 7f2b5d64a640 1 -- 172.21.15.6:0/3348337575 >> v1:172.21.15.6:6790/0 conn(0x7f2b580b4ff0 legacy=0x7f2b5813a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:15.714 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.713+0000 7f2b5d64a640 1 -- 172.21.15.6:0/3348337575 shutdown_connections 2024-03-25T19:16:15.714 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.713+0000 7f2b5d64a640 1 -- 172.21.15.6:0/3348337575 >> 172.21.15.6:0/3348337575 conn(0x7f2b580617a0 msgr2=0x7f2b58145420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:16:15.714 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.713+0000 7f2b5d64a640 1 -- 172.21.15.6:0/3348337575 shutdown_connections 2024-03-25T19:16:15.714 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.713+0000 7f2b5d64a640 1 -- 172.21.15.6:0/3348337575 wait complete. 2024-03-25T19:16:15.714 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:15.713+0000 7f2b5d64a640 1 librados: shutdown 2024-03-25T19:16:15.726 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:16:22.509 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:02:39.855 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:16:22.515 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:02:39.862 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:16:22.533 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:16:22.554 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:02:39.901 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:16:22.559 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:02:39.876 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:16:22.572 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:02:39.890 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:16:22.600 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:02:39.917 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:16:22.612 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:02:39.959 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:16:22.626 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:02:39.973 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:16:22.641 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.641+0000 7f30d051e640 10 monclient: get_monmap_and_config 2024-03-25T19:16:22.641 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.641+0000 7f30d051e640 10 monclient: build_initial_monmap 2024-03-25T19:16:22.641 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.641+0000 7f30d051e640 10 monclient: monmap: 2024-03-25T19:16:22.641 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:16:22.642 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:22.642 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:16:22.642641+0000 2024-03-25T19:16:22.642 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:16:22.642641+0000 2024-03-25T19:16:22.642 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:16:22.642 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:22.642 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:16:22.642 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:16:22.642 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:16:22.642 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:22.643 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.642+0000 7f30d051e640 1 Processor -- start 2024-03-25T19:16:22.644 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.642+0000 7f30d051e640 1 -- start start 2024-03-25T19:16:22.644 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.642+0000 7f30d051e640 10 monclient: init 2024-03-25T19:16:22.644 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.643+0000 7f30d051e640 10 monclient: _reopen_session rank -1 2024-03-25T19:16:22.644 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.643+0000 7f30d051e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:16:22.644 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.643+0000 7f30d051e640 10 monclient(hunting): picked mon.noname-b con 0x7f30c80b4e70 addr v1:172.21.15.6:6790/0 2024-03-25T19:16:22.644 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.643+0000 7f30d051e640 10 monclient(hunting): picked mon.noname-a con 0x7f30c80b44b0 addr v1:172.21.15.6:6789/0 2024-03-25T19:16:22.644 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.643+0000 7f30d051e640 10 monclient(hunting): picked mon.noname-c con 0x7f30c80b1b00 addr v1:172.21.15.192:6789/0 2024-03-25T19:16:22.644 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.643+0000 7f30d051e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c80a96f0 con 0x7f30c80b44b0 2024-03-25T19:16:22.644 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.643+0000 7f30d051e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c80ab0e0 con 0x7f30c80b1b00 2024-03-25T19:16:22.644 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.643+0000 7f30d051e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c8145f70 con 0x7f30c80b4e70 2024-03-25T19:16:22.644 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.643+0000 7f30d051e640 10 monclient(hunting): _renew_subs 2024-03-25T19:16:22.644 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.643+0000 7f30d051e640 10 monclient(hunting): authenticate will time out at 6086.883089s 2024-03-25T19:16:22.659 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:02:40.006 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:16:22.665 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.665+0000 7f30cdab9640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f30c80b44b0 0x7f30c8145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51354/0 (socket says 172.21.15.6:51354) 2024-03-25T19:16:22.665 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.665+0000 7f30cdab9640 1 -- 172.21.15.6:0/1421868508 learned_addr learned my addr 172.21.15.6:0/1421868508 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:16:22.685 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.685+0000 7f30cd2b8640 1 -- 172.21.15.6:0/1421868508 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 661762189 0 0) 0x7f30c80a96f0 con 0x7f30c80b44b0 2024-03-25T19:16:22.685 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.685+0000 7f30cd2b8640 10 monclient(hunting): _negotiate 2024-03-25T19:16:22.685 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.685+0000 7f30cd2b8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:16:22.685 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.685+0000 7f30cd2b8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:16:22.685 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.685+0000 7f30cd2b8640 10 monclient(hunting): my global_id is 4320 2024-03-25T19:16:22.685 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.685+0000 7f30cd2b8640 1 -- 172.21.15.6:0/1421868508 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30b4003520 con 0x7f30c80b44b0 2024-03-25T19:16:22.696 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.696+0000 7f30cd2b8640 1 -- 172.21.15.6:0/1421868508 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 544023046 0 0) 0x7f30b4003520 con 0x7f30c80b44b0 2024-03-25T19:16:22.696 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.696+0000 7f30cd2b8640 1 -- 172.21.15.6:0/1421868508 >> v1:172.21.15.6:6790/0 conn(0x7f30c80b4e70 legacy=0x7f30c813f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:22.696 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.696+0000 7f30cd2b8640 1 -- 172.21.15.6:0/1421868508 >> v1:172.21.15.192:6789/0 conn(0x7f30c80b1b00 legacy=0x7f30c814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:22.696 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.696+0000 7f30cd2b8640 10 monclient: _finish_hunting 0 2024-03-25T19:16:22.696 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.696+0000 7f30cd2b8640 1 monclient: found mon.noname-a 2024-03-25T19:16:22.696 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.696+0000 7f30cd2b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:16:22.696 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.696+0000 7f30cd2b8640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-03-25T19:16:22.697 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.696+0000 7f30cd2b8640 1 -- 172.21.15.6:0/1421868508 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c80a9b80 con 0x7f30c80b44b0 2024-03-25T19:16:22.697 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.696+0000 7f30cd2b8640 10 monclient: _finish_auth 0 2024-03-25T19:16:22.697 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.696+0000 7f30cd2b8640 10 monclient: _check_auth_tickets 2024-03-25T19:16:22.697 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.696+0000 7f30cd2b8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:16:22.697 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.696+0000 7f30cd2b8640 1 -- 172.21.15.6:0/1421868508 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f30b8001690 con 0x7f30c80b44b0 2024-03-25T19:16:22.697 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.696+0000 7f30cd2b8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:16:22.697 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.696+0000 7f30cd2b8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:16:22.697 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.696+0000 7f30cd2b8640 10 monclient: dump: 2024-03-25T19:16:22.697 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:16:22.697 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:22.697 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:22.697 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:22.697 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:16:22.697 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:22.697 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:16:22.698 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:16:22.698 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:16:22.698 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:22.698 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.696+0000 7f30d051e640 5 monclient: authenticate success, global_id 4320 2024-03-25T19:16:22.698 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.696+0000 7f30d051e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:16:22.707 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.707+0000 7f30cd2b8640 1 -- 172.21.15.6:0/1421868508 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30b80017f0 con 0x7f30c80b44b0 2024-03-25T19:16:22.708 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.707+0000 7f30cd2b8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:16:22.708 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.707+0000 7f30cd2b8640 1 -- 172.21.15.6:0/1421868508 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f30b8001c40 con 0x7f30c80b44b0 2024-03-25T19:16:22.708 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.707+0000 7f30cd2b8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:16:22.708 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.708+0000 7f30cd2b8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:16:22.708 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.708+0000 7f30cd2b8640 10 monclient: dump: 2024-03-25T19:16:22.708 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:16:22.708 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:22.708 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:22.708 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:22.708 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:16:22.708 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:22.708 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:16:22.708 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:16:22.708 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:16:22.708 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:22.709 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.708+0000 7f30d051e640 10 monclient: get_monmap_and_config success 2024-03-25T19:16:22.709 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.708+0000 7f30d051e640 4 set_mon_vals no callback set 2024-03-25T19:16:22.710 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.708+0000 7f30d051e640 10 monclient: shutdown 2024-03-25T19:16:22.710 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.708+0000 7f30d051e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:16:22.710 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.708+0000 7f30d051e640 1 -- 172.21.15.6:0/1421868508 >> v1:172.21.15.6:6789/0 conn(0x7f30c80b44b0 legacy=0x7f30c8145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:22.710 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.709+0000 7f30d051e640 1 -- 172.21.15.6:0/1421868508 shutdown_connections 2024-03-25T19:16:22.710 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.709+0000 7f30d051e640 1 -- 172.21.15.6:0/1421868508 >> 172.21.15.6:0/1421868508 conn(0x7f30c80617a0 msgr2=0x7f30c8061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:16:22.710 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.709+0000 7f30d051e640 1 -- 172.21.15.6:0/1421868508 shutdown_connections 2024-03-25T19:16:22.710 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.709+0000 7f30d051e640 1 -- 172.21.15.6:0/1421868508 wait complete. 2024-03-25T19:16:22.711 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.710+0000 7f30d051e640 10 monclient: build_initial_monmap 2024-03-25T19:16:22.711 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.710+0000 7f30d051e640 10 monclient: monmap: 2024-03-25T19:16:22.711 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:16:22.711 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:16:22.711 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:16:22.711 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:16:22.711 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:16:22.711 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:16:22.711 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:16:22.711 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:16:22.711 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:16:22.711 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:22.711 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.710+0000 7f30d051e640 1 librados: starting msgr at 2024-03-25T19:16:22.711 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.710+0000 7f30d051e640 1 librados: starting objecter 2024-03-25T19:16:22.711 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.710+0000 7f30d051e640 1 Processor -- start 2024-03-25T19:16:22.712 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.710+0000 7f30d051e640 1 -- start start 2024-03-25T19:16:22.712 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.710+0000 7f30d051e640 1 librados: setting wanted keys 2024-03-25T19:16:22.712 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.710+0000 7f30d051e640 1 librados: calling monclient init 2024-03-25T19:16:22.712 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.710+0000 7f30d051e640 10 monclient: init 2024-03-25T19:16:22.713 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.711+0000 7f30d051e640 10 monclient: _reopen_session rank -1 2024-03-25T19:16:22.713 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.711+0000 7f30d051e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:16:22.713 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.711+0000 7f30d051e640 10 monclient(hunting): picked mon.noname-c con 0x7f30c80b1b00 addr v1:172.21.15.6:6790/0 2024-03-25T19:16:22.713 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.711+0000 7f30d051e640 10 monclient(hunting): picked mon.noname-a con 0x7f30c80b44b0 addr v1:172.21.15.6:6789/0 2024-03-25T19:16:22.713 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.711+0000 7f30d051e640 10 monclient(hunting): picked mon.noname-b con 0x7f30c80b4e70 addr v1:172.21.15.192:6789/0 2024-03-25T19:16:22.713 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.711+0000 7f30d051e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c813a8b0 con 0x7f30c80b44b0 2024-03-25T19:16:22.713 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.711+0000 7f30d051e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c813c470 con 0x7f30c80b4e70 2024-03-25T19:16:22.713 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.711+0000 7f30d051e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c813c750 con 0x7f30c80b1b00 2024-03-25T19:16:22.713 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.711+0000 7f30d051e640 10 monclient(hunting): _renew_subs 2024-03-25T19:16:22.713 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.711+0000 7f30d051e640 10 monclient(hunting): authenticate will time out at 6086.951280s 2024-03-25T19:16:22.720 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.719+0000 7f30cdab9640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f30c80b44b0 0x7f30c813ae70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51384/0 (socket says 172.21.15.6:51384) 2024-03-25T19:16:22.720 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.719+0000 7f30cdab9640 1 -- 172.21.15.6:0/1229819320 learned_addr learned my addr 172.21.15.6:0/1229819320 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:16:22.720 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.720+0000 7f30beffd640 10 client.?.objecter ms_handle_connect 0x7f30c80b44b0 2024-03-25T19:16:22.720 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.720+0000 7f30beffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:16:22.725 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.726+0000 7f30beffd640 10 client.?.objecter ms_handle_connect 0x7f30c80b4e70 2024-03-25T19:16:22.726 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.726+0000 7f30beffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:16:22.730 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.730+0000 7f30beffd640 1 -- 172.21.15.6:0/1229819320 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2617914254 0 0) 0x7f30c813c470 con 0x7f30c80b4e70 2024-03-25T19:16:22.730 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.730+0000 7f30beffd640 10 monclient(hunting): _negotiate 2024-03-25T19:16:22.730 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.730+0000 7f30beffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:16:22.731 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.730+0000 7f30beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:16:22.731 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.730+0000 7f30beffd640 10 monclient(hunting): my global_id is 4324 2024-03-25T19:16:22.731 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.730+0000 7f30beffd640 1 -- 172.21.15.6:0/1229819320 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30ac004eb0 con 0x7f30c80b4e70 2024-03-25T19:16:22.733 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.732+0000 7f30beffd640 1 -- 172.21.15.6:0/1229819320 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1997468816 0 0) 0x7f30c813a8b0 con 0x7f30c80b44b0 2024-03-25T19:16:22.733 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.732+0000 7f30beffd640 10 monclient(hunting): _negotiate 2024-03-25T19:16:22.733 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.732+0000 7f30beffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:16:22.733 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.732+0000 7f30beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:16:22.733 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.732+0000 7f30beffd640 10 monclient(hunting): my global_id is 4323 2024-03-25T19:16:22.733 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.732+0000 7f30beffd640 1 -- 172.21.15.6:0/1229819320 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30c813c470 con 0x7f30c80b44b0 2024-03-25T19:16:22.739 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.738+0000 7f30beffd640 1 -- 172.21.15.6:0/1229819320 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 124086523 0 0) 0x7f30ac004eb0 con 0x7f30c80b4e70 2024-03-25T19:16:22.739 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.738+0000 7f30beffd640 1 -- 172.21.15.6:0/1229819320 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30c813a8b0 con 0x7f30c80b4e70 2024-03-25T19:16:22.739 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.738+0000 7f30beffd640 1 -- 172.21.15.6:0/1229819320 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f30c0001c30 con 0x7f30c80b4e70 2024-03-25T19:16:22.739 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.738+0000 7f30beffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:16:22.739 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.738+0000 7f30beffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:16:22.739 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.738+0000 7f30beffd640 10 monclient(hunting): dump: 2024-03-25T19:16:22.739 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:16:22.739 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:22.739 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:22.739 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:22.739 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:16:22.739 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:22.739 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:16:22.740 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:16:22.740 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:16:22.740 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:22.740 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.738+0000 7f30beffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:16:22.750 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.750+0000 7f30beffd640 1 -- 172.21.15.6:0/1229819320 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2385762013 0 0) 0x7f30c813c470 con 0x7f30c80b44b0 2024-03-25T19:16:22.750 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.750+0000 7f30beffd640 1 -- 172.21.15.6:0/1229819320 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30ac004eb0 con 0x7f30c80b44b0 2024-03-25T19:16:22.750 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.750+0000 7f30beffd640 1 -- 172.21.15.6:0/1229819320 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f30b8001b00 con 0x7f30c80b44b0 2024-03-25T19:16:22.751 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.750+0000 7f30beffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:16:22.751 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.750+0000 7f30beffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:16:22.751 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.750+0000 7f30beffd640 10 monclient(hunting): dump: 2024-03-25T19:16:22.751 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:16:22.751 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:22.751 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:22.751 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:22.751 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:16:22.751 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:22.751 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:16:22.751 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:16:22.751 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:16:22.751 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:22.754 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.754+0000 7f30beffd640 1 -- 172.21.15.6:0/1229819320 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1223603101 0 0) 0x7f30c813a8b0 con 0x7f30c80b4e70 2024-03-25T19:16:22.754 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.754+0000 7f30beffd640 1 -- 172.21.15.6:0/1229819320 >> v1:172.21.15.6:6790/0 conn(0x7f30c80b1b00 legacy=0x7f30c8140160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:22.754 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.754+0000 7f30beffd640 1 -- 172.21.15.6:0/1229819320 >> v1:172.21.15.6:6789/0 conn(0x7f30c80b44b0 legacy=0x7f30c813ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:22.755 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.754+0000 7f30beffd640 10 monclient: _finish_hunting 0 2024-03-25T19:16:22.755 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.754+0000 7f30beffd640 1 monclient: found mon.b 2024-03-25T19:16:22.755 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.754+0000 7f30beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:16:22.755 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.754+0000 7f30beffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:16:22.755 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.754+0000 7f30beffd640 1 -- 172.21.15.6:0/1229819320 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c813a3d0 con 0x7f30c80b4e70 2024-03-25T19:16:22.755 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.754+0000 7f30beffd640 10 monclient: _finish_auth 0 2024-03-25T19:16:22.755 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.754+0000 7f30beffd640 10 monclient: _check_auth_tickets 2024-03-25T19:16:22.755 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.754+0000 7f30beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:16:22.755 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.754+0000 7f30d051e640 5 monclient: authenticate success, global_id 4324 2024-03-25T19:16:22.755 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.754+0000 7f30d051e640 10 monclient: _renew_subs 2024-03-25T19:16:22.755 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.754+0000 7f30d051e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:16:22.755 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.754+0000 7f30d051e640 1 -- 172.21.15.6:0/1229819320 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30c813b580 con 0x7f30c80b4e70 2024-03-25T19:16:22.756 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.754+0000 7f30d051e640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:16:22.756 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.754+0000 7f30d051e640 10 monclient: _renew_subs 2024-03-25T19:16:22.756 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.754+0000 7f30d051e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:16:22.756 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.754+0000 7f30d051e640 1 -- 172.21.15.6:0/1229819320 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f30c8144d50 con 0x7f30c80b4e70 2024-03-25T19:16:22.756 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.754+0000 7f30d051e640 1 librados: init done 2024-03-25T19:16:22.756 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.755+0000 7f30d051e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:16:22.756 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.755+0000 7f30d051e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:16:22.756 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.755+0000 7f30d051e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:16:22.757 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:22.755+0000 7f30d051e640 1 -- 172.21.15.6:0/1229819320 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30c813a620 con 0x7f30c80b4e70 2024-03-25T19:16:23.036 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:23.036+0000 7f30beffd640 1 -- 172.21.15.6:0/1229819320 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30c0003970 con 0x7f30c80b4e70 2024-03-25T19:16:23.036 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:23.036+0000 7f30beffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:16:23.036 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:23.036+0000 7f30beffd640 1 -- 172.21.15.6:0/1229819320 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f30c0003d00 con 0x7f30c80b4e70 2024-03-25T19:16:23.036 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:23.036+0000 7f30beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:16:23.036 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:23.036+0000 7f30beffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:16:23.036 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:23.036+0000 7f30beffd640 10 monclient: dump: 2024-03-25T19:16:23.036 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:16:23.036 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:23.037 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:23.037 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:23.037 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:16:23.037 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:23.037 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:16:23.037 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:16:23.037 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:16:23.037 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:23.037 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:23.036+0000 7f30ccab7640 4 set_mon_vals no callback set 2024-03-25T19:16:23.038 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:23.037+0000 7f30beffd640 1 -- 172.21.15.6:0/1229819320 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f30c0003f20 con 0x7f30c80b4e70 2024-03-25T19:16:23.038 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:23.037+0000 7f30ce2ba640 10 monclient: get_auth_request con 0x7f30ac06dec0 auth_method 0 2024-03-25T19:16:23.038 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:23.037+0000 7f30beffd640 10 client.4324.objecter ms_handle_connect 0x7f30ac06dec0 2024-03-25T19:16:23.042 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:23.042+0000 7f30beffd640 1 -- 172.21.15.6:0/1229819320 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f30c0053800 con 0x7f30c80b4e70 2024-03-25T19:16:23.042 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:23.042+0000 7f30beffd640 10 client.4324.objecter ms_dispatch 0x7f30c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:16:23.042 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:23.042+0000 7f30beffd640 3 client.4324.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:16:23.043 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:23.042+0000 7f30beffd640 3 client.4324.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:16:23.043 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:23.042+0000 7f30beffd640 20 client.4324.objecter dump_active .. 0 homeless 2024-03-25T19:16:23.043 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:23.043+0000 7f30beffd640 1 -- 172.21.15.6:0/1229819320 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f30c004f310 con 0x7f30c80b4e70 2024-03-25T19:16:23.043 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:23.043+0000 7f30beffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:16:23.043 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:23.043+0000 7f30beffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:16:23.294 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:23.294+0000 7f30d051e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:16:23.294 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:23.294+0000 7f30d051e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:16:23.294 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:23.294+0000 7f30d051e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:16:23.294 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:23.294+0000 7f30d051e640 1 -- 172.21.15.6:0/1229819320 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30c804f730 con 0x7f30c80b4e70 2024-03-25T19:16:23.326 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:23.326+0000 7f30beffd640 1 -- 172.21.15.6:0/1229819320 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f30c0051240 con 0x7f30c80b4e70 2024-03-25T19:16:23.326 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:23.326+0000 7f30beffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:16:23.326 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:23.326+0000 7f30beffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:16:23.327 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:16:23.327 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:16:23.329 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:23.329+0000 7f30d051e640 10 librados: watch_flush enter 2024-03-25T19:16:23.330 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:23.329+0000 7f30d051e640 10 librados: watch_flush exit 2024-03-25T19:16:23.330 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:23.329+0000 7f30d051e640 20 client.4324.objecter shutdown clearing up homeless session... 2024-03-25T19:16:23.330 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:23.329+0000 7f30d051e640 10 client.4324.objecter successfully canceled tick 2024-03-25T19:16:23.330 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:23.329+0000 7f30d051e640 1 -- 172.21.15.6:0/1229819320 >> v1:172.21.15.6:6800/2936652650 conn(0x7f30ac06dec0 legacy=0x7f30ac070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:23.330 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:23.329+0000 7f30d051e640 10 monclient: shutdown 2024-03-25T19:16:23.330 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:23.329+0000 7f30d051e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:16:23.330 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:23.329+0000 7f30d051e640 1 -- 172.21.15.6:0/1229819320 >> v1:172.21.15.192:6789/0 conn(0x7f30c80b4e70 legacy=0x7f30c81e25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:23.330 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:23.329+0000 7f30d051e640 1 -- 172.21.15.6:0/1229819320 shutdown_connections 2024-03-25T19:16:23.330 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:23.329+0000 7f30d051e640 1 -- 172.21.15.6:0/1229819320 >> 172.21.15.6:0/1229819320 conn(0x7f30c80617a0 msgr2=0x7f30c80b6510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:16:23.330 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:23.329+0000 7f30d051e640 1 -- 172.21.15.6:0/1229819320 shutdown_connections 2024-03-25T19:16:23.331 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:23.329+0000 7f30d051e640 1 -- 172.21.15.6:0/1229819320 wait complete. 2024-03-25T19:16:23.331 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:23.329+0000 7f30d051e640 1 librados: shutdown 2024-03-25T19:16:23.340 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:16:30.146 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:16:30.266 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.265+0000 7f5a4a97b640 10 monclient: get_monmap_and_config 2024-03-25T19:16:30.266 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.265+0000 7f5a4a97b640 10 monclient: build_initial_monmap 2024-03-25T19:16:30.266 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.265+0000 7f5a4a97b640 10 monclient: monmap: 2024-03-25T19:16:30.266 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:16:30.266 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:30.266 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:16:30.266877+0000 2024-03-25T19:16:30.266 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:16:30.266877+0000 2024-03-25T19:16:30.266 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:16:30.266 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:30.267 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:16:30.267 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:16:30.267 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:16:30.267 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:30.268 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.266+0000 7f5a4a97b640 1 Processor -- start 2024-03-25T19:16:30.268 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.266+0000 7f5a4a97b640 1 -- start start 2024-03-25T19:16:30.268 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.266+0000 7f5a4a97b640 10 monclient: init 2024-03-25T19:16:30.268 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.267+0000 7f5a4a97b640 10 monclient: _reopen_session rank -1 2024-03-25T19:16:30.268 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.267+0000 7f5a4a97b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:16:30.268 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.267+0000 7f5a4a97b640 10 monclient(hunting): picked mon.noname-c con 0x7f5a440b4ff0 addr v1:172.21.15.192:6789/0 2024-03-25T19:16:30.268 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.267+0000 7f5a4a97b640 10 monclient(hunting): picked mon.noname-b con 0x7f5a440b4670 addr v1:172.21.15.6:6790/0 2024-03-25T19:16:30.268 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.267+0000 7f5a4a97b640 10 monclient(hunting): picked mon.noname-a con 0x7f5a4413c480 addr v1:172.21.15.6:6789/0 2024-03-25T19:16:30.268 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.267+0000 7f5a4a97b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a440ab0e0 con 0x7f5a4413c480 2024-03-25T19:16:30.268 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.267+0000 7f5a4a97b640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a440ab3c0 con 0x7f5a440b4ff0 2024-03-25T19:16:30.268 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.267+0000 7f5a4a97b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a440a9f30 con 0x7f5a440b4670 2024-03-25T19:16:30.268 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.267+0000 7f5a4a97b640 10 monclient(hunting): _renew_subs 2024-03-25T19:16:30.269 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.267+0000 7f5a4a97b640 10 monclient(hunting): authenticate will time out at 6094.507491s 2024-03-25T19:16:30.274 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.274+0000 7f5a437fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f5a440b4670 0x7f5a441425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:38942/0 (socket says 172.21.15.6:38942) 2024-03-25T19:16:30.274 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.274+0000 7f5a437fe640 1 -- 172.21.15.6:0/3618038316 learned_addr learned my addr 172.21.15.6:0/3618038316 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:16:30.291 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.291+0000 7f5a42ffd640 1 -- 172.21.15.6:0/3618038316 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3270817805 0 0) 0x7f5a440a9f30 con 0x7f5a440b4670 2024-03-25T19:16:30.291 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.291+0000 7f5a42ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:16:30.291 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.291+0000 7f5a42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:16:30.291 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.291+0000 7f5a42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:16:30.291 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.291+0000 7f5a42ffd640 10 monclient(hunting): my global_id is 4325 2024-03-25T19:16:30.291 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.291+0000 7f5a42ffd640 1 -- 172.21.15.6:0/3618038316 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a2c003520 con 0x7f5a440b4670 2024-03-25T19:16:30.296 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.296+0000 7f5a42ffd640 1 -- 172.21.15.6:0/3618038316 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4014438100 0 0) 0x7f5a440ab3c0 con 0x7f5a440b4ff0 2024-03-25T19:16:30.296 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.296+0000 7f5a42ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:16:30.296 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.296+0000 7f5a42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:16:30.296 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.296+0000 7f5a42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:16:30.296 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.296+0000 7f5a42ffd640 10 monclient(hunting): my global_id is 4327 2024-03-25T19:16:30.296 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.296+0000 7f5a42ffd640 1 -- 172.21.15.6:0/3618038316 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a440a9f30 con 0x7f5a440b4ff0 2024-03-25T19:16:30.305 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.305+0000 7f5a42ffd640 1 -- 172.21.15.6:0/3618038316 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 863177431 0 0) 0x7f5a2c003520 con 0x7f5a440b4670 2024-03-25T19:16:30.306 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.305+0000 7f5a42ffd640 1 -- 172.21.15.6:0/3618038316 >> v1:172.21.15.192:6789/0 conn(0x7f5a440b4ff0 legacy=0x7f5a4413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:30.306 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.305+0000 7f5a42ffd640 1 -- 172.21.15.6:0/3618038316 >> v1:172.21.15.6:6789/0 conn(0x7f5a4413c480 legacy=0x7f5a4414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:30.306 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.305+0000 7f5a42ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:16:30.306 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.305+0000 7f5a42ffd640 1 monclient: found mon.noname-b 2024-03-25T19:16:30.306 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.305+0000 7f5a42ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:16:30.306 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.305+0000 7f5a42ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-03-25T19:16:30.306 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.305+0000 7f5a42ffd640 1 -- 172.21.15.6:0/3618038316 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a440a9b70 con 0x7f5a440b4670 2024-03-25T19:16:30.306 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.305+0000 7f5a42ffd640 10 monclient: _finish_auth 0 2024-03-25T19:16:30.306 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.305+0000 7f5a42ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:16:30.306 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.305+0000 7f5a42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:16:30.306 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.305+0000 7f5a42ffd640 1 -- 172.21.15.6:0/3618038316 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f5a38001690 con 0x7f5a440b4670 2024-03-25T19:16:30.306 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.305+0000 7f5a42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:16:30.306 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.305+0000 7f5a42ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:16:30.307 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.305+0000 7f5a42ffd640 10 monclient: dump: 2024-03-25T19:16:30.307 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:16:30.307 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:30.307 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:30.307 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:30.307 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:16:30.307 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:30.307 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:16:30.307 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:16:30.307 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:16:30.307 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:30.307 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.305+0000 7f5a4a97b640 5 monclient: authenticate success, global_id 4325 2024-03-25T19:16:30.307 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.305+0000 7f5a4a97b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:16:30.322 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.322+0000 7f5a42ffd640 1 -- 172.21.15.6:0/3618038316 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a380017f0 con 0x7f5a440b4670 2024-03-25T19:16:30.322 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.322+0000 7f5a42ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:16:30.322 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.322+0000 7f5a42ffd640 1 -- 172.21.15.6:0/3618038316 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f5a38001c00 con 0x7f5a440b4670 2024-03-25T19:16:30.322 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.322+0000 7f5a4a97b640 10 monclient: get_monmap_and_config success 2024-03-25T19:16:30.322 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.322+0000 7f5a4a97b640 4 set_mon_vals no callback set 2024-03-25T19:16:30.322 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.322+0000 7f5a42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:16:30.323 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.322+0000 7f5a42ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:16:30.323 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.322+0000 7f5a42ffd640 10 monclient: dump: 2024-03-25T19:16:30.323 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:16:30.323 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:30.323 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:30.323 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:30.323 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:16:30.323 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:30.323 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:16:30.323 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:16:30.323 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:16:30.323 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:30.325 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.323+0000 7f5a4a97b640 10 monclient: shutdown 2024-03-25T19:16:30.325 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.323+0000 7f5a4a97b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:16:30.325 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.323+0000 7f5a4a97b640 1 -- 172.21.15.6:0/3618038316 >> v1:172.21.15.6:6790/0 conn(0x7f5a440b4670 legacy=0x7f5a441425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:30.325 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.323+0000 7f5a4a97b640 1 -- 172.21.15.6:0/3618038316 shutdown_connections 2024-03-25T19:16:30.325 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.323+0000 7f5a4a97b640 1 -- 172.21.15.6:0/3618038316 >> 172.21.15.6:0/3618038316 conn(0x7f5a440617a0 msgr2=0x7f5a44061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:16:30.325 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.323+0000 7f5a4a97b640 1 -- 172.21.15.6:0/3618038316 shutdown_connections 2024-03-25T19:16:30.325 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.323+0000 7f5a4a97b640 1 -- 172.21.15.6:0/3618038316 wait complete. 2024-03-25T19:16:30.325 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.324+0000 7f5a4a97b640 10 monclient: build_initial_monmap 2024-03-25T19:16:30.325 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.324+0000 7f5a4a97b640 10 monclient: monmap: 2024-03-25T19:16:30.325 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:16:30.325 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:16:30.326 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:16:30.326 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:16:30.326 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:16:30.326 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:16:30.326 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:16:30.326 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:16:30.326 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:16:30.326 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:30.326 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.324+0000 7f5a4a97b640 1 librados: starting msgr at 2024-03-25T19:16:30.326 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.324+0000 7f5a4a97b640 1 librados: starting objecter 2024-03-25T19:16:30.326 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.325+0000 7f5a4a97b640 1 Processor -- start 2024-03-25T19:16:30.326 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.325+0000 7f5a4a97b640 1 -- start start 2024-03-25T19:16:30.326 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.325+0000 7f5a4a97b640 1 librados: setting wanted keys 2024-03-25T19:16:30.326 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.325+0000 7f5a4a97b640 1 librados: calling monclient init 2024-03-25T19:16:30.327 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.325+0000 7f5a4a97b640 10 monclient: init 2024-03-25T19:16:30.327 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.326+0000 7f5a4a97b640 10 monclient: _reopen_session rank -1 2024-03-25T19:16:30.327 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.326+0000 7f5a4a97b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:16:30.327 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.326+0000 7f5a4a97b640 10 monclient(hunting): picked mon.noname-c con 0x7f5a440b4670 addr v1:172.21.15.6:6790/0 2024-03-25T19:16:30.327 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.326+0000 7f5a4a97b640 10 monclient(hunting): picked mon.noname-b con 0x7f5a440b4ff0 addr v1:172.21.15.192:6789/0 2024-03-25T19:16:30.327 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.326+0000 7f5a4a97b640 10 monclient(hunting): picked mon.noname-a con 0x7f5a4413c480 addr v1:172.21.15.6:6789/0 2024-03-25T19:16:30.327 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.326+0000 7f5a4a97b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a4413b120 con 0x7f5a4413c480 2024-03-25T19:16:30.327 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.326+0000 7f5a4a97b640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a440b1990 con 0x7f5a440b4ff0 2024-03-25T19:16:30.327 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.326+0000 7f5a4a97b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a440b0130 con 0x7f5a440b4670 2024-03-25T19:16:30.328 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.326+0000 7f5a4a97b640 10 monclient(hunting): _renew_subs 2024-03-25T19:16:30.328 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.326+0000 7f5a4a97b640 10 monclient(hunting): authenticate will time out at 6094.566000s 2024-03-25T19:16:30.330 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.330+0000 7f5a437fe640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f5a440b4ff0 0x7f5a440ae390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:55374/0 (socket says 172.21.15.6:55374) 2024-03-25T19:16:30.330 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.330+0000 7f5a437fe640 1 -- 172.21.15.6:0/351106913 learned_addr learned my addr 172.21.15.6:0/351106913 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:16:30.332 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.332+0000 7f5a40ff9640 10 client.?.objecter ms_handle_connect 0x7f5a440b4ff0 2024-03-25T19:16:30.332 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.332+0000 7f5a40ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:16:30.335 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.335+0000 7f5a40ff9640 10 client.?.objecter ms_handle_connect 0x7f5a4413c480 2024-03-25T19:16:30.335 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.335+0000 7f5a40ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:16:30.335 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.335+0000 7f5a40ff9640 10 client.?.objecter ms_handle_connect 0x7f5a440b4670 2024-03-25T19:16:30.335 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.335+0000 7f5a40ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:16:30.341 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.341+0000 7f5a40ff9640 1 -- 172.21.15.6:0/351106913 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1551292298 0 0) 0x7f5a440b1990 con 0x7f5a440b4ff0 2024-03-25T19:16:30.341 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.341+0000 7f5a40ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:16:30.341 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.341+0000 7f5a40ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:16:30.341 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.341+0000 7f5a40ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:16:30.341 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.341+0000 7f5a40ff9640 10 monclient(hunting): my global_id is 4330 2024-03-25T19:16:30.341 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.341+0000 7f5a40ff9640 1 -- 172.21.15.6:0/351106913 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a30004eb0 con 0x7f5a440b4ff0 2024-03-25T19:16:30.350 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.350+0000 7f5a40ff9640 1 -- 172.21.15.6:0/351106913 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2834446445 0 0) 0x7f5a440b0130 con 0x7f5a440b4670 2024-03-25T19:16:30.350 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.350+0000 7f5a40ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:16:30.351 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.350+0000 7f5a40ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:16:30.351 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.350+0000 7f5a40ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:16:30.351 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.350+0000 7f5a40ff9640 10 monclient(hunting): my global_id is 4328 2024-03-25T19:16:30.351 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.350+0000 7f5a40ff9640 1 -- 172.21.15.6:0/351106913 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a440b1990 con 0x7f5a440b4670 2024-03-25T19:16:30.353 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.353+0000 7f5a40ff9640 1 -- 172.21.15.6:0/351106913 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2932975289 0 0) 0x7f5a4413b120 con 0x7f5a4413c480 2024-03-25T19:16:30.353 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.353+0000 7f5a40ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:16:30.353 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.353+0000 7f5a40ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:16:30.353 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.353+0000 7f5a40ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:16:30.353 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.353+0000 7f5a40ff9640 10 monclient(hunting): my global_id is 4329 2024-03-25T19:16:30.353 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.353+0000 7f5a40ff9640 1 -- 172.21.15.6:0/351106913 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a440b0130 con 0x7f5a4413c480 2024-03-25T19:16:30.361 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.361+0000 7f5a40ff9640 1 -- 172.21.15.6:0/351106913 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3216932768 0 0) 0x7f5a440b1990 con 0x7f5a440b4670 2024-03-25T19:16:30.361 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.361+0000 7f5a40ff9640 1 -- 172.21.15.6:0/351106913 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a4413b120 con 0x7f5a440b4670 2024-03-25T19:16:30.361 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.361+0000 7f5a40ff9640 1 -- 172.21.15.6:0/351106913 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f5a34001930 con 0x7f5a440b4670 2024-03-25T19:16:30.361 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.361+0000 7f5a40ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:16:30.361 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.361+0000 7f5a40ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:16:30.361 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.361+0000 7f5a40ff9640 10 monclient(hunting): dump: 2024-03-25T19:16:30.361 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:16:30.361 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:30.361 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:30.361 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:30.361 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:16:30.361 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:30.361 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:16:30.362 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:16:30.362 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:16:30.362 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:30.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.361+0000 7f5a40ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:16:30.365 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.365+0000 7f5a40ff9640 1 -- 172.21.15.6:0/351106913 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1690924081 0 0) 0x7f5a440b0130 con 0x7f5a4413c480 2024-03-25T19:16:30.365 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.365+0000 7f5a40ff9640 1 -- 172.21.15.6:0/351106913 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a440b1990 con 0x7f5a4413c480 2024-03-25T19:16:30.365 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.365+0000 7f5a40ff9640 1 -- 172.21.15.6:0/351106913 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f5a3c001a60 con 0x7f5a4413c480 2024-03-25T19:16:30.365 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.365+0000 7f5a40ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:16:30.365 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.365+0000 7f5a40ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:16:30.365 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.365+0000 7f5a40ff9640 10 monclient(hunting): dump: 2024-03-25T19:16:30.366 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:16:30.366 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:30.366 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:30.366 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:30.366 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:16:30.366 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:30.366 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:16:30.366 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:16:30.366 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:16:30.366 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:30.367 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.366+0000 7f5a40ff9640 1 -- 172.21.15.6:0/351106913 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 417587522 0 0) 0x7f5a30004eb0 con 0x7f5a440b4ff0 2024-03-25T19:16:30.367 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.366+0000 7f5a40ff9640 1 -- 172.21.15.6:0/351106913 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a440b0130 con 0x7f5a440b4ff0 2024-03-25T19:16:30.367 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.366+0000 7f5a40ff9640 1 -- 172.21.15.6:0/351106913 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f5a38001c40 con 0x7f5a440b4ff0 2024-03-25T19:16:30.367 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.366+0000 7f5a40ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:16:30.367 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.366+0000 7f5a40ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:16:30.367 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.366+0000 7f5a40ff9640 10 monclient(hunting): dump: 2024-03-25T19:16:30.367 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:16:30.367 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:30.368 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:30.368 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:30.368 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:16:30.368 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:30.368 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:16:30.368 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:16:30.368 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:16:30.368 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:30.369 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.368+0000 7f5a40ff9640 1 -- 172.21.15.6:0/351106913 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1565244810 0 0) 0x7f5a4413b120 con 0x7f5a440b4670 2024-03-25T19:16:30.369 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.368+0000 7f5a40ff9640 1 -- 172.21.15.6:0/351106913 >> v1:172.21.15.192:6789/0 conn(0x7f5a440b4ff0 legacy=0x7f5a440ae390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:30.369 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.368+0000 7f5a40ff9640 1 -- 172.21.15.6:0/351106913 >> v1:172.21.15.6:6789/0 conn(0x7f5a4413c480 legacy=0x7f5a440aeaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:30.369 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.368+0000 7f5a40ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:16:30.369 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.368+0000 7f5a40ff9640 1 monclient: found mon.c 2024-03-25T19:16:30.369 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.368+0000 7f5a40ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:16:30.369 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.368+0000 7f5a40ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:16:30.370 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.368+0000 7f5a40ff9640 1 -- 172.21.15.6:0/351106913 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a440b0410 con 0x7f5a440b4670 2024-03-25T19:16:30.370 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.368+0000 7f5a40ff9640 10 monclient: _finish_auth 0 2024-03-25T19:16:30.370 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.368+0000 7f5a40ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:16:30.370 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.368+0000 7f5a40ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:16:30.370 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.368+0000 7f5a4a97b640 5 monclient: authenticate success, global_id 4328 2024-03-25T19:16:30.370 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.368+0000 7f5a4a97b640 10 monclient: _renew_subs 2024-03-25T19:16:30.370 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.368+0000 7f5a4a97b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:16:30.370 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.368+0000 7f5a4a97b640 1 -- 172.21.15.6:0/351106913 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a440b06f0 con 0x7f5a440b4670 2024-03-25T19:16:30.370 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.368+0000 7f5a4a97b640 10 client.4328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:16:30.370 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.368+0000 7f5a4a97b640 10 monclient: _renew_subs 2024-03-25T19:16:30.370 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.368+0000 7f5a4a97b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:16:30.370 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.368+0000 7f5a4a97b640 1 -- 172.21.15.6:0/351106913 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a441e35a0 con 0x7f5a440b4670 2024-03-25T19:16:30.370 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.368+0000 7f5a4a97b640 1 librados: init done 2024-03-25T19:16:30.371 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.370+0000 7f5a4a97b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:16:30.371 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.370+0000 7f5a4a97b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:16:30.371 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.370+0000 7f5a4a97b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:16:30.371 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.370+0000 7f5a4a97b640 1 -- 172.21.15.6:0/351106913 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a4406dbd0 con 0x7f5a440b4670 2024-03-25T19:16:30.678 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.678+0000 7f5a40ff9640 1 -- 172.21.15.6:0/351106913 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a34003610 con 0x7f5a440b4670 2024-03-25T19:16:30.679 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.678+0000 7f5a40ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:16:30.679 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.678+0000 7f5a40ff9640 1 -- 172.21.15.6:0/351106913 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f5a34003920 con 0x7f5a440b4670 2024-03-25T19:16:30.679 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.678+0000 7f5a40ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:16:30.679 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.678+0000 7f5a40ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:16:30.679 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.678+0000 7f5a40ff9640 10 monclient: dump: 2024-03-25T19:16:30.679 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:16:30.679 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:30.679 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:30.679 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:30.679 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:16:30.679 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:30.680 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:16:30.680 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:16:30.680 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:16:30.680 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:30.680 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.678+0000 7f5a427fc640 4 set_mon_vals no callback set 2024-03-25T19:16:30.680 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.679+0000 7f5a40ff9640 1 -- 172.21.15.6:0/351106913 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f5a340198f0 con 0x7f5a440b4670 2024-03-25T19:16:30.680 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.679+0000 7f5a437fe640 10 monclient: get_auth_request con 0x7f5a3006e0f0 auth_method 0 2024-03-25T19:16:30.680 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.680+0000 7f5a40ff9640 10 client.4328.objecter ms_handle_connect 0x7f5a3006e0f0 2024-03-25T19:16:30.686 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.686+0000 7f5a40ff9640 1 -- 172.21.15.6:0/351106913 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f5a34053550 con 0x7f5a440b4670 2024-03-25T19:16:30.686 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.686+0000 7f5a40ff9640 10 client.4328.objecter ms_dispatch 0x7f5a44000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:16:30.687 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.686+0000 7f5a40ff9640 3 client.4328.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:16:30.687 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.686+0000 7f5a40ff9640 3 client.4328.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:16:30.687 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.686+0000 7f5a40ff9640 20 client.4328.objecter dump_active .. 0 homeless 2024-03-25T19:16:30.687 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.686+0000 7f5a40ff9640 1 -- 172.21.15.6:0/351106913 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5a3404f060 con 0x7f5a440b4670 2024-03-25T19:16:30.687 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.686+0000 7f5a40ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:16:30.687 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.686+0000 7f5a40ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:16:30.937 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.937+0000 7f5a4a97b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:16:30.937 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.937+0000 7f5a4a97b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:16:30.938 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.937+0000 7f5a4a97b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:16:30.938 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:30.937+0000 7f5a4a97b640 1 -- 172.21.15.6:0/351106913 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a440ae0b0 con 0x7f5a440b4670 2024-03-25T19:16:31.052 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:31.052+0000 7f5a40ff9640 1 -- 172.21.15.6:0/351106913 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f5a440ae0b0 con 0x7f5a440b4670 2024-03-25T19:16:31.052 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:31.052+0000 7f5a40ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:16:31.052 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:31.052+0000 7f5a40ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:16:31.052 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:16:31.052 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:16:31.054 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:31.054+0000 7f5a4a97b640 10 librados: watch_flush enter 2024-03-25T19:16:31.054 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:31.054+0000 7f5a4a97b640 10 librados: watch_flush exit 2024-03-25T19:16:31.055 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:31.054+0000 7f5a4a97b640 20 client.4328.objecter shutdown clearing up homeless session... 2024-03-25T19:16:31.055 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:31.054+0000 7f5a4a97b640 10 client.4328.objecter successfully canceled tick 2024-03-25T19:16:31.055 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:31.054+0000 7f5a4a97b640 1 -- 172.21.15.6:0/351106913 >> v1:172.21.15.6:6800/2936652650 conn(0x7f5a3006e0f0 legacy=0x7f5a300705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:31.055 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:31.054+0000 7f5a4a97b640 10 monclient: shutdown 2024-03-25T19:16:31.055 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:31.054+0000 7f5a4a97b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:16:31.055 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:31.054+0000 7f5a4a97b640 1 -- 172.21.15.6:0/351106913 >> v1:172.21.15.6:6790/0 conn(0x7f5a440b4670 legacy=0x7f5a440b0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:31.056 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:31.055+0000 7f5a4a97b640 1 -- 172.21.15.6:0/351106913 shutdown_connections 2024-03-25T19:16:31.056 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:31.055+0000 7f5a4a97b640 1 -- 172.21.15.6:0/351106913 >> 172.21.15.6:0/351106913 conn(0x7f5a440617a0 msgr2=0x7f5a44145670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:16:31.056 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:31.055+0000 7f5a4a97b640 1 -- 172.21.15.6:0/351106913 shutdown_connections 2024-03-25T19:16:31.056 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:31.055+0000 7f5a4a97b640 1 -- 172.21.15.6:0/351106913 wait complete. 2024-03-25T19:16:31.056 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:31.055+0000 7f5a4a97b640 1 librados: shutdown 2024-03-25T19:16:31.067 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:16:37.874 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:16:37.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:37.988+0000 7f4a072ba640 10 monclient: get_monmap_and_config 2024-03-25T19:16:37.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:37.988+0000 7f4a072ba640 10 monclient: build_initial_monmap 2024-03-25T19:16:37.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:37.988+0000 7f4a072ba640 10 monclient: monmap: 2024-03-25T19:16:37.988 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:16:37.988 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:37.988 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:16:37.989369+0000 2024-03-25T19:16:37.989 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:16:37.989369+0000 2024-03-25T19:16:37.989 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:16:37.989 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:37.989 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:16:37.989 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:16:37.989 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:16:37.989 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:37.990 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:37.989+0000 7f4a072ba640 1 Processor -- start 2024-03-25T19:16:37.990 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:37.989+0000 7f4a072ba640 1 -- start start 2024-03-25T19:16:37.990 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:37.989+0000 7f4a072ba640 10 monclient: init 2024-03-25T19:16:37.991 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:37.990+0000 7f4a072ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:16:37.991 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:37.990+0000 7f4a072ba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:16:37.991 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:37.990+0000 7f4a072ba640 10 monclient(hunting): picked mon.noname-b con 0x7f4a000b4ff0 addr v1:172.21.15.6:6790/0 2024-03-25T19:16:37.991 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:37.990+0000 7f4a072ba640 10 monclient(hunting): picked mon.noname-c con 0x7f4a000b4670 addr v1:172.21.15.192:6789/0 2024-03-25T19:16:37.991 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:37.990+0000 7f4a072ba640 10 monclient(hunting): picked mon.noname-a con 0x7f4a0013c480 addr v1:172.21.15.6:6789/0 2024-03-25T19:16:37.991 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:37.990+0000 7f4a072ba640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a000ab0e0 con 0x7f4a0013c480 2024-03-25T19:16:37.991 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:37.990+0000 7f4a072ba640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a000ab3c0 con 0x7f4a000b4670 2024-03-25T19:16:37.991 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:37.990+0000 7f4a072ba640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a000a9f30 con 0x7f4a000b4ff0 2024-03-25T19:16:37.991 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:37.990+0000 7f4a072ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:16:37.991 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:37.990+0000 7f4a072ba640 10 monclient(hunting): authenticate will time out at 6102.230112s 2024-03-25T19:16:37.995 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:37.995+0000 7f4a04855640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f4a000b4670 0x7f4a001425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:59898/0 (socket says 172.21.15.6:59898) 2024-03-25T19:16:37.995 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:37.995+0000 7f4a04855640 1 -- 172.21.15.6:0/1705581609 learned_addr learned my addr 172.21.15.6:0/1705581609 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:16:38.004 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.004+0000 7f49f7fff640 1 -- 172.21.15.6:0/1705581609 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3923423174 0 0) 0x7f4a000ab3c0 con 0x7f4a000b4670 2024-03-25T19:16:38.004 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.004+0000 7f49f7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:16:38.004 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.004+0000 7f49f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:16:38.004 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.004+0000 7f49f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:16:38.004 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.004+0000 7f49f7fff640 10 monclient(hunting): my global_id is 4333 2024-03-25T19:16:38.004 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.004+0000 7f49f7fff640 1 -- 172.21.15.6:0/1705581609 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49ec003520 con 0x7f4a000b4670 2024-03-25T19:16:38.006 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.006+0000 7f49f7fff640 1 -- 172.21.15.6:0/1705581609 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2206983320 0 0) 0x7f4a000a9f30 con 0x7f4a000b4ff0 2024-03-25T19:16:38.006 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.006+0000 7f49f7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:16:38.006 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.006+0000 7f49f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:16:38.006 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.006+0000 7f49f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:16:38.006 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.006+0000 7f49f7fff640 10 monclient(hunting): my global_id is 4331 2024-03-25T19:16:38.007 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.006+0000 7f49f7fff640 1 -- 172.21.15.6:0/1705581609 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a000ab3c0 con 0x7f4a000b4ff0 2024-03-25T19:16:38.012 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.012+0000 7f49f7fff640 1 -- 172.21.15.6:0/1705581609 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1857095499 0 0) 0x7f49ec003520 con 0x7f4a000b4670 2024-03-25T19:16:38.012 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.012+0000 7f49f7fff640 1 -- 172.21.15.6:0/1705581609 >> v1:172.21.15.6:6790/0 conn(0x7f4a000b4ff0 legacy=0x7f4a0013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:38.012 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.012+0000 7f49f7fff640 1 -- 172.21.15.6:0/1705581609 >> v1:172.21.15.6:6789/0 conn(0x7f4a0013c480 legacy=0x7f4a0014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:16:38.012 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.012+0000 7f49f7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:16:38.012 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.012+0000 7f49f7fff640 1 monclient: found mon.noname-c 2024-03-25T19:16:38.012 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.012+0000 7f49f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:16:38.012 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.012+0000 7f49f7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:16:38.012 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.012+0000 7f49f7fff640 1 -- 172.21.15.6:0/1705581609 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a000a9b70 con 0x7f4a000b4670 2024-03-25T19:16:38.012 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.012+0000 7f49f7fff640 10 monclient: _finish_auth 0 2024-03-25T19:16:38.013 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.012+0000 7f49f7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:16:38.013 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.012+0000 7f49f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:16:38.013 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.012+0000 7f49f7fff640 1 -- 172.21.15.6:0/1705581609 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f49f0001690 con 0x7f4a000b4670 2024-03-25T19:16:38.013 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.012+0000 7f4a072ba640 5 monclient: authenticate success, global_id 4333 2024-03-25T19:16:38.013 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.012+0000 7f4a072ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:16:38.013 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.012+0000 7f49f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:16:38.013 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.012+0000 7f49f7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:16:38.013 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.012+0000 7f49f7fff640 10 monclient: dump: 2024-03-25T19:16:38.013 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:16:38.013 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:38.013 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:38.013 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:38.013 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:16:38.014 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:38.014 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:16:38.014 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:16:38.014 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:16:38.014 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:38.014 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.012+0000 7f4a072ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:16:38.019 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.019+0000 7f49f7fff640 1 -- 172.21.15.6:0/1705581609 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49f00017f0 con 0x7f4a000b4670 2024-03-25T19:16:38.019 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.019+0000 7f49f7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:16:38.019 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.019+0000 7f49f7fff640 1 -- 172.21.15.6:0/1705581609 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f49f0001c00 con 0x7f4a000b4670 2024-03-25T19:16:38.019 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.019+0000 7f4a072ba640 10 monclient: get_monmap_and_config success 2024-03-25T19:16:38.019 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.019+0000 7f4a072ba640 4 set_mon_vals no callback set 2024-03-25T19:16:38.020 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.019+0000 7f49f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:16:38.020 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.019+0000 7f49f7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:16:38.020 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.019+0000 7f49f7fff640 10 monclient: dump: 2024-03-25T19:16:38.020 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:16:38.020 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:38.020 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:38.020 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:38.020 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:16:38.020 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:38.020 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:16:38.020 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:16:38.020 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:16:38.020 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:38.021 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.020+0000 7f4a072ba640 10 monclient: shutdown 2024-03-25T19:16:38.021 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.020+0000 7f4a072ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:16:38.021 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.020+0000 7f4a072ba640 1 -- 172.21.15.6:0/1705581609 >> v1:172.21.15.192:6789/0 conn(0x7f4a000b4670 legacy=0x7f4a001425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:38.021 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.020+0000 7f4a072ba640 1 -- 172.21.15.6:0/1705581609 shutdown_connections 2024-03-25T19:16:38.021 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.020+0000 7f4a072ba640 1 -- 172.21.15.6:0/1705581609 >> 172.21.15.6:0/1705581609 conn(0x7f4a000617a0 msgr2=0x7f4a00061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:16:38.022 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.021+0000 7f4a072ba640 1 -- 172.21.15.6:0/1705581609 shutdown_connections 2024-03-25T19:16:38.022 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.021+0000 7f4a072ba640 1 -- 172.21.15.6:0/1705581609 wait complete. 2024-03-25T19:16:38.023 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.022+0000 7f4a072ba640 10 monclient: build_initial_monmap 2024-03-25T19:16:38.023 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.022+0000 7f4a072ba640 10 monclient: monmap: 2024-03-25T19:16:38.023 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:16:38.023 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:16:38.024 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:16:38.024 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:16:38.024 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:16:38.024 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:16:38.024 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:16:38.024 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:16:38.024 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:16:38.024 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:38.024 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.022+0000 7f4a072ba640 1 librados: starting msgr at 2024-03-25T19:16:38.024 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.022+0000 7f4a072ba640 1 librados: starting objecter 2024-03-25T19:16:38.024 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.022+0000 7f4a072ba640 1 Processor -- start 2024-03-25T19:16:38.024 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.022+0000 7f4a072ba640 1 -- start start 2024-03-25T19:16:38.024 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.022+0000 7f4a072ba640 1 librados: setting wanted keys 2024-03-25T19:16:38.024 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.022+0000 7f4a072ba640 1 librados: calling monclient init 2024-03-25T19:16:38.024 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.022+0000 7f4a072ba640 10 monclient: init 2024-03-25T19:16:38.025 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.023+0000 7f4a072ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:16:38.025 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.023+0000 7f4a072ba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:16:38.025 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.023+0000 7f4a072ba640 10 monclient(hunting): picked mon.noname-c con 0x7f4a000b4670 addr v1:172.21.15.6:6790/0 2024-03-25T19:16:38.025 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.023+0000 7f4a072ba640 10 monclient(hunting): picked mon.noname-a con 0x7f4a000b4ff0 addr v1:172.21.15.6:6789/0 2024-03-25T19:16:38.025 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.023+0000 7f4a072ba640 10 monclient(hunting): picked mon.noname-b con 0x7f4a0013c480 addr v1:172.21.15.192:6789/0 2024-03-25T19:16:38.025 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.023+0000 7f4a072ba640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a00130c10 con 0x7f4a000b4ff0 2024-03-25T19:16:38.025 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.023+0000 7f4a072ba640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a00144800 con 0x7f4a0013c480 2024-03-25T19:16:38.025 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.023+0000 7f4a072ba640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a00132fb0 con 0x7f4a000b4670 2024-03-25T19:16:38.025 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.024+0000 7f4a072ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:16:38.025 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.024+0000 7f4a072ba640 10 monclient(hunting): authenticate will time out at 6102.263787s 2024-03-25T19:16:38.038 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.038+0000 7f4a05857640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f4a0013c480 0x7f4a0012fbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:59928/0 (socket says 172.21.15.6:59928) 2024-03-25T19:16:38.038 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.038+0000 7f4a05857640 1 -- 172.21.15.6:0/1475659585 learned_addr learned my addr 172.21.15.6:0/1475659585 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:16:38.040 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.041+0000 7f49f5ffb640 10 client.?.objecter ms_handle_connect 0x7f4a0013c480 2024-03-25T19:16:38.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.041+0000 7f49f5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:16:38.050 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.050+0000 7f49f5ffb640 10 client.?.objecter ms_handle_connect 0x7f4a000b4ff0 2024-03-25T19:16:38.050 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.050+0000 7f49f5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:16:38.057 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.057+0000 7f49f5ffb640 10 client.?.objecter ms_handle_connect 0x7f4a000b4670 2024-03-25T19:16:38.057 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.057+0000 7f49f5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:16:38.064 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.064+0000 7f49f5ffb640 1 -- 172.21.15.6:0/1475659585 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4268603546 0 0) 0x7f4a00144800 con 0x7f4a0013c480 2024-03-25T19:16:38.064 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.064+0000 7f49f5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:16:38.064 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.064+0000 7f49f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:16:38.064 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.064+0000 7f49f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:16:38.064 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.064+0000 7f49f5ffb640 10 monclient(hunting): my global_id is 4336 2024-03-25T19:16:38.064 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.064+0000 7f49f5ffb640 1 -- 172.21.15.6:0/1475659585 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49e4004eb0 con 0x7f4a0013c480 2024-03-25T19:16:38.069 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.069+0000 7f49f5ffb640 1 -- 172.21.15.6:0/1475659585 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1162660760 0 0) 0x7f4a00130c10 con 0x7f4a000b4ff0 2024-03-25T19:16:38.069 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.069+0000 7f49f5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:16:38.069 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.069+0000 7f49f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:16:38.069 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.069+0000 7f49f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:16:38.069 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.069+0000 7f49f5ffb640 10 monclient(hunting): my global_id is 4332 2024-03-25T19:16:38.069 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.069+0000 7f49f5ffb640 1 -- 172.21.15.6:0/1475659585 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a00144800 con 0x7f4a000b4ff0 2024-03-25T19:16:38.081 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.081+0000 7f49f5ffb640 1 -- 172.21.15.6:0/1475659585 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1488414030 0 0) 0x7f4a00132fb0 con 0x7f4a000b4670 2024-03-25T19:16:38.081 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.081+0000 7f49f5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:16:38.081 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.081+0000 7f49f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:16:38.081 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.081+0000 7f49f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:16:38.082 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.081+0000 7f49f5ffb640 10 monclient(hunting): my global_id is 4334 2024-03-25T19:16:38.082 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.081+0000 7f49f5ffb640 1 -- 172.21.15.6:0/1475659585 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a00130c10 con 0x7f4a000b4670 2024-03-25T19:16:38.085 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.085+0000 7f49f5ffb640 1 -- 172.21.15.6:0/1475659585 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3201452675 0 0) 0x7f49e4004eb0 con 0x7f4a0013c480 2024-03-25T19:16:38.085 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.085+0000 7f49f5ffb640 1 -- 172.21.15.6:0/1475659585 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a00132fb0 con 0x7f4a0013c480 2024-03-25T19:16:38.085 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.085+0000 7f49f5ffb640 1 -- 172.21.15.6:0/1475659585 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f49f8001ad0 con 0x7f4a0013c480 2024-03-25T19:16:38.085 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.085+0000 7f49f5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:16:38.085 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.085+0000 7f49f5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:16:38.085 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.085+0000 7f49f5ffb640 10 monclient(hunting): dump: 2024-03-25T19:16:38.085 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:16:38.085 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:38.086 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:38.086 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:38.086 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:16:38.086 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:38.086 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:16:38.086 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:16:38.086 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:16:38.086 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:38.086 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.085+0000 7f49f5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:16:38.093 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.093+0000 7f49f5ffb640 1 -- 172.21.15.6:0/1475659585 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3187257108 0 0) 0x7f4a00144800 con 0x7f4a000b4ff0 2024-03-25T19:16:38.093 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.093+0000 7f49f5ffb640 1 -- 172.21.15.6:0/1475659585 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49e4004eb0 con 0x7f4a000b4ff0 2024-03-25T19:16:38.093 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.093+0000 7f49f5ffb640 1 -- 172.21.15.6:0/1475659585 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f49f0001b00 con 0x7f4a000b4ff0 2024-03-25T19:16:38.093 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.093+0000 7f49f5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:16:38.093 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.093+0000 7f49f5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:16:38.093 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.093+0000 7f49f5ffb640 10 monclient(hunting): dump: 2024-03-25T19:16:38.093 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:16:38.093 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:38.093 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:38.093 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:38.094 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:16:38.094 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:38.094 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:16:38.094 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:16:38.094 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:16:38.094 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:38.109 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.108+0000 7f49f5ffb640 1 -- 172.21.15.6:0/1475659585 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3683758339 0 0) 0x7f4a00132fb0 con 0x7f4a0013c480 2024-03-25T19:16:38.109 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.108+0000 7f49f5ffb640 1 -- 172.21.15.6:0/1475659585 >> v1:172.21.15.6:6790/0 conn(0x7f4a000b4670 legacy=0x7f4a0013c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:38.109 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.108+0000 7f49f5ffb640 1 -- 172.21.15.6:0/1475659585 >> v1:172.21.15.6:6789/0 conn(0x7f4a000b4ff0 legacy=0x7f4a00145f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:38.109 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.108+0000 7f49f5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:16:38.109 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.108+0000 7f49f5ffb640 1 monclient: found mon.b 2024-03-25T19:16:38.109 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.108+0000 7f49f5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:16:38.109 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.108+0000 7f49f5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:16:38.109 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.108+0000 7f49f5ffb640 1 -- 172.21.15.6:0/1475659585 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a0013eaa0 con 0x7f4a0013c480 2024-03-25T19:16:38.110 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.108+0000 7f49f5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:16:38.110 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.108+0000 7f49f5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:16:38.110 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.108+0000 7f49f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:16:38.110 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.108+0000 7f4a072ba640 5 monclient: authenticate success, global_id 4336 2024-03-25T19:16:38.110 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.108+0000 7f4a072ba640 10 monclient: _renew_subs 2024-03-25T19:16:38.110 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.108+0000 7f4a072ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:16:38.110 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.108+0000 7f4a072ba640 1 -- 172.21.15.6:0/1475659585 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a00143960 con 0x7f4a0013c480 2024-03-25T19:16:38.110 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.109+0000 7f4a072ba640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:16:38.110 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.109+0000 7f4a072ba640 10 monclient: _renew_subs 2024-03-25T19:16:38.110 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.109+0000 7f4a072ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:16:38.110 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.109+0000 7f4a072ba640 1 -- 172.21.15.6:0/1475659585 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a0013f7f0 con 0x7f4a0013c480 2024-03-25T19:16:38.110 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.109+0000 7f4a072ba640 1 librados: init done 2024-03-25T19:16:38.113 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.110+0000 7f4a072ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:16:38.113 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.110+0000 7f4a072ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:16:38.113 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.110+0000 7f4a072ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:16:38.113 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.110+0000 7f4a072ba640 1 -- 172.21.15.6:0/1475659585 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a0013bf10 con 0x7f4a0013c480 2024-03-25T19:16:38.388 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.388+0000 7f49f5ffb640 1 -- 172.21.15.6:0/1475659585 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49f8001fe0 con 0x7f4a0013c480 2024-03-25T19:16:38.388 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.388+0000 7f49f5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:16:38.388 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.388+0000 7f49f5ffb640 1 -- 172.21.15.6:0/1475659585 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f49f8004d80 con 0x7f4a0013c480 2024-03-25T19:16:38.389 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.388+0000 7f49f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:16:38.389 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.388+0000 7f49f5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:16:38.389 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.388+0000 7f49f5ffb640 10 monclient: dump: 2024-03-25T19:16:38.389 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:16:38.389 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:38.389 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:38.389 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:38.389 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:16:38.389 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:38.389 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:16:38.390 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:16:38.390 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:16:38.390 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:38.390 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.388+0000 7f49f77fe640 4 set_mon_vals no callback set 2024-03-25T19:16:38.390 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.389+0000 7f49f5ffb640 1 -- 172.21.15.6:0/1475659585 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f49f801ad50 con 0x7f4a0013c480 2024-03-25T19:16:38.390 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.390+0000 7f4a05056640 10 monclient: get_auth_request con 0x7f49e406dec0 auth_method 0 2024-03-25T19:16:38.390 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.390+0000 7f49f5ffb640 10 client.4336.objecter ms_handle_connect 0x7f49e406dec0 2024-03-25T19:16:38.396 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.395+0000 7f49f5ffb640 1 -- 172.21.15.6:0/1475659585 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f49f8054a00 con 0x7f4a0013c480 2024-03-25T19:16:38.396 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.395+0000 7f49f5ffb640 10 client.4336.objecter ms_dispatch 0x7f4a00000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:16:38.396 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.395+0000 7f49f5ffb640 3 client.4336.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:16:38.396 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.395+0000 7f49f5ffb640 3 client.4336.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:16:38.396 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.395+0000 7f49f5ffb640 20 client.4336.objecter dump_active .. 0 homeless 2024-03-25T19:16:38.396 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.395+0000 7f49f5ffb640 1 -- 172.21.15.6:0/1475659585 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f49f8050340 con 0x7f4a0013c480 2024-03-25T19:16:38.396 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.395+0000 7f49f5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:16:38.396 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.395+0000 7f49f5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:16:38.632 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.632+0000 7f4a072ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:16:38.632 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.632+0000 7f4a072ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:16:38.632 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.632+0000 7f4a072ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:16:38.632 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.632+0000 7f4a072ba640 1 -- 172.21.15.6:0/1475659585 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4a00143af0 con 0x7f4a0013c480 2024-03-25T19:16:38.693 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.693+0000 7f49f5ffb640 1 -- 172.21.15.6:0/1475659585 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f49f8052440 con 0x7f4a0013c480 2024-03-25T19:16:38.693 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.693+0000 7f49f5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:16:38.693 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.693+0000 7f49f5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:16:38.693 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:16:38.693 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:16:38.696 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.695+0000 7f4a072ba640 10 librados: watch_flush enter 2024-03-25T19:16:38.696 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.695+0000 7f4a072ba640 10 librados: watch_flush exit 2024-03-25T19:16:38.696 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.695+0000 7f4a072ba640 20 client.4336.objecter shutdown clearing up homeless session... 2024-03-25T19:16:38.696 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.695+0000 7f4a072ba640 10 client.4336.objecter successfully canceled tick 2024-03-25T19:16:38.696 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.695+0000 7f4a072ba640 1 -- 172.21.15.6:0/1475659585 >> v1:172.21.15.6:6800/2936652650 conn(0x7f49e406dec0 legacy=0x7f49e4070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:38.696 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.695+0000 7f4a072ba640 10 monclient: shutdown 2024-03-25T19:16:38.696 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.695+0000 7f4a072ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:16:38.696 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.695+0000 7f4a072ba640 1 -- 172.21.15.6:0/1475659585 >> v1:172.21.15.192:6789/0 conn(0x7f4a0013c480 legacy=0x7f4a0012fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:38.696 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.695+0000 7f4a072ba640 1 -- 172.21.15.6:0/1475659585 shutdown_connections 2024-03-25T19:16:38.696 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.695+0000 7f4a072ba640 1 -- 172.21.15.6:0/1475659585 >> 172.21.15.6:0/1475659585 conn(0x7f4a000617a0 msgr2=0x7f4a000b8690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:16:38.697 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.696+0000 7f4a072ba640 1 -- 172.21.15.6:0/1475659585 shutdown_connections 2024-03-25T19:16:38.697 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.696+0000 7f4a072ba640 1 -- 172.21.15.6:0/1475659585 wait complete. 2024-03-25T19:16:38.697 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:38.696+0000 7f4a072ba640 1 librados: shutdown 2024-03-25T19:16:38.707 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:16:42.602 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:02:59.948 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:16:42.614 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:02:59.961 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:16:42.647 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:02:59.994 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:16:42.654 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:02:59.971 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:16:42.658 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:03:00.005 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:16:42.662 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:02:59.980 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:16:42.664 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:03:00.011 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:16:42.688 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:03:00.035 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:16:42.694 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:03:00.012 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:16:45.514 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:16:45.627 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.628+0000 7fcc56025640 10 monclient: get_monmap_and_config 2024-03-25T19:16:45.628 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.628+0000 7fcc56025640 10 monclient: build_initial_monmap 2024-03-25T19:16:45.628 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.628+0000 7fcc56025640 10 monclient: monmap: 2024-03-25T19:16:45.628 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:16:45.628 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:45.628 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:16:45.629105+0000 2024-03-25T19:16:45.628 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:16:45.629105+0000 2024-03-25T19:16:45.628 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:16:45.628 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:45.628 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:16:45.628 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:16:45.628 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:16:45.628 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:45.630 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.629+0000 7fcc56025640 1 Processor -- start 2024-03-25T19:16:45.630 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.629+0000 7fcc56025640 1 -- start start 2024-03-25T19:16:45.630 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.629+0000 7fcc56025640 10 monclient: init 2024-03-25T19:16:45.630 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.629+0000 7fcc56025640 10 monclient: _reopen_session rank -1 2024-03-25T19:16:45.630 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.629+0000 7fcc56025640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:16:45.630 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.629+0000 7fcc56025640 10 monclient(hunting): picked mon.noname-c con 0x7fcc500b4e70 addr v1:172.21.15.192:6789/0 2024-03-25T19:16:45.630 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.629+0000 7fcc56025640 10 monclient(hunting): picked mon.noname-a con 0x7fcc500b44b0 addr v1:172.21.15.6:6789/0 2024-03-25T19:16:45.630 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.629+0000 7fcc56025640 10 monclient(hunting): picked mon.noname-b con 0x7fcc500b1b00 addr v1:172.21.15.6:6790/0 2024-03-25T19:16:45.630 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.629+0000 7fcc56025640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc500a96f0 con 0x7fcc500b44b0 2024-03-25T19:16:45.631 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.630+0000 7fcc56025640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc500ab0e0 con 0x7fcc500b4e70 2024-03-25T19:16:45.631 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.630+0000 7fcc56025640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc50145f70 con 0x7fcc500b1b00 2024-03-25T19:16:45.631 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.630+0000 7fcc56025640 10 monclient(hunting): _renew_subs 2024-03-25T19:16:45.631 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.630+0000 7fcc56025640 10 monclient(hunting): authenticate will time out at 6109.869811s 2024-03-25T19:16:45.638 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.638+0000 7fcc4effd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fcc500b44b0 0x7fcc50145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56514/0 (socket says 172.21.15.6:56514) 2024-03-25T19:16:45.638 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.638+0000 7fcc4effd640 1 -- 172.21.15.6:0/1916286839 learned_addr learned my addr 172.21.15.6:0/1916286839 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:16:45.651 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.651+0000 7fcc4e7fc640 1 -- 172.21.15.6:0/1916286839 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3538819915 0 0) 0x7fcc500a96f0 con 0x7fcc500b44b0 2024-03-25T19:16:45.651 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.651+0000 7fcc4e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:16:45.652 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.651+0000 7fcc4e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:16:45.652 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.651+0000 7fcc4e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:16:45.652 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.651+0000 7fcc4e7fc640 10 monclient(hunting): my global_id is 4335 2024-03-25T19:16:45.652 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.651+0000 7fcc4e7fc640 1 -- 172.21.15.6:0/1916286839 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc38003520 con 0x7fcc500b44b0 2024-03-25T19:16:45.665 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.663+0000 7fcc4e7fc640 1 -- 172.21.15.6:0/1916286839 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4075686232 0 0) 0x7fcc500ab0e0 con 0x7fcc500b4e70 2024-03-25T19:16:45.665 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.663+0000 7fcc4e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:16:45.666 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.663+0000 7fcc4e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:16:45.666 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.663+0000 7fcc4e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:16:45.666 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.663+0000 7fcc4e7fc640 10 monclient(hunting): my global_id is 4339 2024-03-25T19:16:45.666 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.663+0000 7fcc4e7fc640 1 -- 172.21.15.6:0/1916286839 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc500a96f0 con 0x7fcc500b4e70 2024-03-25T19:16:45.666 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.664+0000 7fcc4e7fc640 1 -- 172.21.15.6:0/1916286839 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2530166022 0 0) 0x7fcc50145f70 con 0x7fcc500b1b00 2024-03-25T19:16:45.666 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.664+0000 7fcc4e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:16:45.666 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.664+0000 7fcc4e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:16:45.666 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.664+0000 7fcc4e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:16:45.666 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.664+0000 7fcc4e7fc640 10 monclient(hunting): my global_id is 4337 2024-03-25T19:16:45.666 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.664+0000 7fcc4e7fc640 1 -- 172.21.15.6:0/1916286839 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc500ab0e0 con 0x7fcc500b1b00 2024-03-25T19:16:45.672 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.672+0000 7fcc4e7fc640 1 -- 172.21.15.6:0/1916286839 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3594124249 0 0) 0x7fcc38003520 con 0x7fcc500b44b0 2024-03-25T19:16:45.673 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.672+0000 7fcc4e7fc640 1 -- 172.21.15.6:0/1916286839 >> v1:172.21.15.6:6790/0 conn(0x7fcc500b1b00 legacy=0x7fcc5014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:45.673 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.672+0000 7fcc4e7fc640 1 -- 172.21.15.6:0/1916286839 >> v1:172.21.15.192:6789/0 conn(0x7fcc500b4e70 legacy=0x7fcc5013f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:45.673 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.672+0000 7fcc4e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:16:45.673 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.672+0000 7fcc4e7fc640 1 monclient: found mon.noname-a 2024-03-25T19:16:45.673 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.672+0000 7fcc4e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:16:45.673 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.672+0000 7fcc4e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-03-25T19:16:45.673 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.672+0000 7fcc4e7fc640 1 -- 172.21.15.6:0/1916286839 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc500a9b80 con 0x7fcc500b44b0 2024-03-25T19:16:45.673 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.672+0000 7fcc4e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:16:45.673 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.672+0000 7fcc4e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:16:45.673 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.672+0000 7fcc4e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:16:45.673 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.672+0000 7fcc4e7fc640 1 -- 172.21.15.6:0/1916286839 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fcc44001690 con 0x7fcc500b44b0 2024-03-25T19:16:45.673 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.672+0000 7fcc56025640 5 monclient: authenticate success, global_id 4335 2024-03-25T19:16:45.673 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.672+0000 7fcc56025640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:16:45.673 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.672+0000 7fcc4e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:16:45.674 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.672+0000 7fcc4e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:16:45.674 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.672+0000 7fcc4e7fc640 10 monclient: dump: 2024-03-25T19:16:45.674 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:16:45.674 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:45.674 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:45.674 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:45.674 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:16:45.674 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:45.674 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:16:45.674 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:16:45.674 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:16:45.674 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:45.674 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.672+0000 7fcc56025640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:16:45.676 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.676+0000 7fcc4e7fc640 1 -- 172.21.15.6:0/1916286839 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc440017f0 con 0x7fcc500b44b0 2024-03-25T19:16:45.676 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.676+0000 7fcc4e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:16:45.676 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.676+0000 7fcc4e7fc640 1 -- 172.21.15.6:0/1916286839 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fcc44001c00 con 0x7fcc500b44b0 2024-03-25T19:16:45.676 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.676+0000 7fcc56025640 10 monclient: get_monmap_and_config success 2024-03-25T19:16:45.676 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.676+0000 7fcc56025640 4 set_mon_vals no callback set 2024-03-25T19:16:45.676 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.676+0000 7fcc4e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:16:45.676 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.676+0000 7fcc4e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:16:45.677 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.676+0000 7fcc4e7fc640 10 monclient: dump: 2024-03-25T19:16:45.677 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:16:45.677 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:45.677 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:45.677 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:45.677 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:16:45.677 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:45.677 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:16:45.677 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:16:45.677 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:16:45.677 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:45.679 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.677+0000 7fcc56025640 10 monclient: shutdown 2024-03-25T19:16:45.679 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.677+0000 7fcc56025640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:16:45.679 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.677+0000 7fcc56025640 1 -- 172.21.15.6:0/1916286839 >> v1:172.21.15.6:6789/0 conn(0x7fcc500b44b0 legacy=0x7fcc50145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:45.679 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.677+0000 7fcc56025640 1 -- 172.21.15.6:0/1916286839 shutdown_connections 2024-03-25T19:16:45.679 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.677+0000 7fcc56025640 1 -- 172.21.15.6:0/1916286839 >> 172.21.15.6:0/1916286839 conn(0x7fcc500617a0 msgr2=0x7fcc50061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:16:45.679 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.677+0000 7fcc56025640 1 -- 172.21.15.6:0/1916286839 shutdown_connections 2024-03-25T19:16:45.679 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.677+0000 7fcc56025640 1 -- 172.21.15.6:0/1916286839 wait complete. 2024-03-25T19:16:45.680 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.677+0000 7fcc56025640 10 monclient: build_initial_monmap 2024-03-25T19:16:45.680 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.678+0000 7fcc56025640 10 monclient: monmap: 2024-03-25T19:16:45.680 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:16:45.680 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:16:45.680 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:16:45.680 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:16:45.680 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:16:45.680 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:16:45.680 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:16:45.680 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:16:45.680 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:16:45.680 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:45.680 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.678+0000 7fcc56025640 1 librados: starting msgr at 2024-03-25T19:16:45.680 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.678+0000 7fcc56025640 1 librados: starting objecter 2024-03-25T19:16:45.680 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.678+0000 7fcc56025640 1 Processor -- start 2024-03-25T19:16:45.681 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.678+0000 7fcc56025640 1 -- start start 2024-03-25T19:16:45.681 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.678+0000 7fcc56025640 1 librados: setting wanted keys 2024-03-25T19:16:45.681 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.678+0000 7fcc56025640 1 librados: calling monclient init 2024-03-25T19:16:45.681 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.678+0000 7fcc56025640 10 monclient: init 2024-03-25T19:16:45.681 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.679+0000 7fcc56025640 10 monclient: _reopen_session rank -1 2024-03-25T19:16:45.681 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.679+0000 7fcc56025640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:16:45.681 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.679+0000 7fcc56025640 10 monclient(hunting): picked mon.noname-a con 0x7fcc500b1b00 addr v1:172.21.15.6:6789/0 2024-03-25T19:16:45.681 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.679+0000 7fcc56025640 10 monclient(hunting): picked mon.noname-b con 0x7fcc500b44b0 addr v1:172.21.15.192:6789/0 2024-03-25T19:16:45.681 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.679+0000 7fcc56025640 10 monclient(hunting): picked mon.noname-c con 0x7fcc500b4e70 addr v1:172.21.15.6:6790/0 2024-03-25T19:16:45.681 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.679+0000 7fcc56025640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc501404b0 con 0x7fcc500b1b00 2024-03-25T19:16:45.681 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.679+0000 7fcc56025640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc501435d0 con 0x7fcc500b44b0 2024-03-25T19:16:45.681 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.679+0000 7fcc56025640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc50140160 con 0x7fcc500b4e70 2024-03-25T19:16:45.681 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.679+0000 7fcc56025640 10 monclient(hunting): _renew_subs 2024-03-25T19:16:45.681 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.679+0000 7fcc56025640 10 monclient(hunting): authenticate will time out at 6109.919353s 2024-03-25T19:16:45.694 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.694+0000 7fcc4ffff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fcc500b4e70 0x7fcc501e2d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:55336/0 (socket says 172.21.15.6:55336) 2024-03-25T19:16:45.694 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.694+0000 7fcc4ffff640 1 -- 172.21.15.6:0/3098579865 learned_addr learned my addr 172.21.15.6:0/3098579865 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:16:45.695 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.695+0000 7fcc37fff640 10 client.?.objecter ms_handle_connect 0x7fcc500b4e70 2024-03-25T19:16:45.695 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.695+0000 7fcc37fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:16:45.696 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.696+0000 7fcc37fff640 10 client.?.objecter ms_handle_connect 0x7fcc500b44b0 2024-03-25T19:16:45.696 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.696+0000 7fcc37fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:16:45.705 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.705+0000 7fcc37fff640 10 client.?.objecter ms_handle_connect 0x7fcc500b1b00 2024-03-25T19:16:45.705 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.705+0000 7fcc37fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:16:45.711 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.711+0000 7fcc37fff640 1 -- 172.21.15.6:0/3098579865 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1016768430 0 0) 0x7fcc501435d0 con 0x7fcc500b44b0 2024-03-25T19:16:45.711 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.711+0000 7fcc37fff640 10 monclient(hunting): _negotiate 2024-03-25T19:16:45.711 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.711+0000 7fcc37fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:16:45.711 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.711+0000 7fcc37fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:16:45.711 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.711+0000 7fcc37fff640 10 monclient(hunting): my global_id is 4342 2024-03-25T19:16:45.711 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.711+0000 7fcc37fff640 1 -- 172.21.15.6:0/3098579865 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc3c004eb0 con 0x7fcc500b44b0 2024-03-25T19:16:45.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.721+0000 7fcc37fff640 1 -- 172.21.15.6:0/3098579865 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 98564592 0 0) 0x7fcc501404b0 con 0x7fcc500b1b00 2024-03-25T19:16:45.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.721+0000 7fcc37fff640 10 monclient(hunting): _negotiate 2024-03-25T19:16:45.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.721+0000 7fcc37fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:16:45.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.721+0000 7fcc37fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:16:45.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.721+0000 7fcc37fff640 10 monclient(hunting): my global_id is 4338 2024-03-25T19:16:45.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.721+0000 7fcc37fff640 1 -- 172.21.15.6:0/3098579865 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc501435d0 con 0x7fcc500b1b00 2024-03-25T19:16:45.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.721+0000 7fcc37fff640 1 -- 172.21.15.6:0/3098579865 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3292874406 0 0) 0x7fcc50140160 con 0x7fcc500b4e70 2024-03-25T19:16:45.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.721+0000 7fcc37fff640 10 monclient(hunting): _negotiate 2024-03-25T19:16:45.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.721+0000 7fcc37fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:16:45.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.721+0000 7fcc37fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:16:45.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.721+0000 7fcc37fff640 10 monclient(hunting): my global_id is 4340 2024-03-25T19:16:45.722 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.721+0000 7fcc37fff640 1 -- 172.21.15.6:0/3098579865 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc501404b0 con 0x7fcc500b4e70 2024-03-25T19:16:45.736 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.736+0000 7fcc37fff640 1 -- 172.21.15.6:0/3098579865 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2727150117 0 0) 0x7fcc501404b0 con 0x7fcc500b4e70 2024-03-25T19:16:45.736 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.736+0000 7fcc37fff640 1 -- 172.21.15.6:0/3098579865 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc50140160 con 0x7fcc500b4e70 2024-03-25T19:16:45.736 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.736+0000 7fcc37fff640 1 -- 172.21.15.6:0/3098579865 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fcc48001c70 con 0x7fcc500b4e70 2024-03-25T19:16:45.736 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.736+0000 7fcc37fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:16:45.736 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.736+0000 7fcc37fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:16:45.736 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.736+0000 7fcc37fff640 10 monclient(hunting): dump: 2024-03-25T19:16:45.737 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:16:45.737 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:45.737 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:45.737 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:45.737 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:16:45.737 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:45.737 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:16:45.737 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:16:45.737 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:16:45.737 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:45.737 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.736+0000 7fcc37fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:16:45.737 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.736+0000 7fcc37fff640 1 -- 172.21.15.6:0/3098579865 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 22419925 0 0) 0x7fcc3c004eb0 con 0x7fcc500b44b0 2024-03-25T19:16:45.737 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.736+0000 7fcc37fff640 1 -- 172.21.15.6:0/3098579865 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc501404b0 con 0x7fcc500b44b0 2024-03-25T19:16:45.737 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.736+0000 7fcc37fff640 1 -- 172.21.15.6:0/3098579865 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fcc44001b00 con 0x7fcc500b44b0 2024-03-25T19:16:45.738 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.736+0000 7fcc37fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:16:45.738 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.736+0000 7fcc37fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:16:45.738 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.736+0000 7fcc37fff640 10 monclient(hunting): dump: 2024-03-25T19:16:45.738 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:16:45.738 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:45.738 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:45.738 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:45.738 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:16:45.738 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:45.738 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:16:45.738 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:16:45.738 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:16:45.738 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:45.739 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.738+0000 7fcc37fff640 1 -- 172.21.15.6:0/3098579865 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2721846218 0 0) 0x7fcc501435d0 con 0x7fcc500b1b00 2024-03-25T19:16:45.739 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.738+0000 7fcc37fff640 1 -- 172.21.15.6:0/3098579865 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc3c004eb0 con 0x7fcc500b1b00 2024-03-25T19:16:45.743 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.743+0000 7fcc37fff640 1 -- 172.21.15.6:0/3098579865 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fcc40001910 con 0x7fcc500b1b00 2024-03-25T19:16:45.743 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.743+0000 7fcc37fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:16:45.743 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.743+0000 7fcc37fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:16:45.743 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.743+0000 7fcc37fff640 10 monclient(hunting): dump: 2024-03-25T19:16:45.743 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:16:45.744 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:45.744 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:45.744 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:45.744 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:16:45.744 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:45.744 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:16:45.744 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:16:45.744 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:16:45.744 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:45.748 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.747+0000 7fcc37fff640 1 -- 172.21.15.6:0/3098579865 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1890158778 0 0) 0x7fcc501404b0 con 0x7fcc500b44b0 2024-03-25T19:16:45.748 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.747+0000 7fcc37fff640 1 -- 172.21.15.6:0/3098579865 >> v1:172.21.15.6:6790/0 conn(0x7fcc500b4e70 legacy=0x7fcc501e2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:45.748 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.747+0000 7fcc37fff640 1 -- 172.21.15.6:0/3098579865 >> v1:172.21.15.6:6789/0 conn(0x7fcc500b1b00 legacy=0x7fcc5013b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:45.748 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.747+0000 7fcc37fff640 10 monclient: _finish_hunting 0 2024-03-25T19:16:45.748 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.747+0000 7fcc37fff640 1 monclient: found mon.b 2024-03-25T19:16:45.748 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.747+0000 7fcc37fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:16:45.748 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.747+0000 7fcc37fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:16:45.748 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.747+0000 7fcc37fff640 1 -- 172.21.15.6:0/3098579865 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc501408d0 con 0x7fcc500b44b0 2024-03-25T19:16:45.748 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.747+0000 7fcc37fff640 10 monclient: _finish_auth 0 2024-03-25T19:16:45.748 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.747+0000 7fcc37fff640 10 monclient: _check_auth_tickets 2024-03-25T19:16:45.748 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.747+0000 7fcc37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:16:45.748 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.747+0000 7fcc56025640 5 monclient: authenticate success, global_id 4342 2024-03-25T19:16:45.749 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.747+0000 7fcc56025640 10 monclient: _renew_subs 2024-03-25T19:16:45.749 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.747+0000 7fcc56025640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:16:45.749 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.747+0000 7fcc56025640 1 -- 172.21.15.6:0/3098579865 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc50142b20 con 0x7fcc500b44b0 2024-03-25T19:16:45.749 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.747+0000 7fcc56025640 10 client.4342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:16:45.749 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.747+0000 7fcc56025640 10 monclient: _renew_subs 2024-03-25T19:16:45.749 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.747+0000 7fcc56025640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:16:45.749 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.747+0000 7fcc56025640 1 -- 172.21.15.6:0/3098579865 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc5013b060 con 0x7fcc500b44b0 2024-03-25T19:16:45.749 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.747+0000 7fcc56025640 1 librados: init done 2024-03-25T19:16:45.750 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.748+0000 7fcc56025640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:16:45.750 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.748+0000 7fcc56025640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:16:45.750 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.748+0000 7fcc56025640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:16:45.750 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.748+0000 7fcc56025640 1 -- 172.21.15.6:0/3098579865 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc50140a80 con 0x7fcc500b44b0 2024-03-25T19:16:45.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.958+0000 7fcc37fff640 1 -- 172.21.15.6:0/3098579865 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc44004a90 con 0x7fcc500b44b0 2024-03-25T19:16:45.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.958+0000 7fcc37fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:16:45.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:45.958+0000 7fcc4dffb640 4 set_mon_vals no callback set 2024-03-25T19:16:46.026 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:46.026+0000 7fcc37fff640 1 -- 172.21.15.6:0/3098579865 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fcc44004c00 con 0x7fcc500b44b0 2024-03-25T19:16:46.026 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:46.026+0000 7fcc37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:16:46.026 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:46.026+0000 7fcc37fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:16:46.027 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:46.026+0000 7fcc37fff640 10 monclient: dump: 2024-03-25T19:16:46.027 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:16:46.027 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:46.027 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:46.027 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:46.027 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:16:46.027 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:46.027 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:16:46.027 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:16:46.027 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:16:46.027 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:46.028 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:46.027+0000 7fcc37fff640 1 -- 172.21.15.6:0/3098579865 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7fcc4401abd0 con 0x7fcc500b44b0 2024-03-25T19:16:46.028 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:46.028+0000 7fcc4f7fe640 10 monclient: get_auth_request con 0x7fcc3c06e0f0 auth_method 0 2024-03-25T19:16:46.028 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:46.028+0000 7fcc37fff640 10 client.4342.objecter ms_handle_connect 0x7fcc3c06e0f0 2024-03-25T19:16:46.029 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:46.029+0000 7fcc37fff640 1 -- 172.21.15.6:0/3098579865 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7fcc44054700 con 0x7fcc500b44b0 2024-03-25T19:16:46.030 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:46.029+0000 7fcc37fff640 10 client.4342.objecter ms_dispatch 0x7fcc50000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:16:46.030 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:46.029+0000 7fcc37fff640 3 client.4342.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:16:46.030 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:46.029+0000 7fcc37fff640 3 client.4342.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:16:46.030 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:46.029+0000 7fcc37fff640 20 client.4342.objecter dump_active .. 0 homeless 2024-03-25T19:16:46.031 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:46.032+0000 7fcc37fff640 1 -- 172.21.15.6:0/3098579865 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcc44050040 con 0x7fcc500b44b0 2024-03-25T19:16:46.031 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:46.032+0000 7fcc37fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:16:46.031 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:46.032+0000 7fcc37fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:16:46.286 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:46.286+0000 7fcc56025640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:16:46.287 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:46.286+0000 7fcc56025640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:16:46.287 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:46.286+0000 7fcc56025640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:16:46.287 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:46.286+0000 7fcc56025640 1 -- 172.21.15.6:0/3098579865 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc50145940 con 0x7fcc500b44b0 2024-03-25T19:16:46.482 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:46.482+0000 7fcc37fff640 1 -- 172.21.15.6:0/3098579865 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7fcc44052140 con 0x7fcc500b44b0 2024-03-25T19:16:46.482 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:46.482+0000 7fcc37fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:16:46.482 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:46.482+0000 7fcc37fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:16:46.482 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:16:46.483 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:16:46.485 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:46.484+0000 7fcc56025640 10 librados: watch_flush enter 2024-03-25T19:16:46.485 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:46.484+0000 7fcc56025640 10 librados: watch_flush exit 2024-03-25T19:16:46.485 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:46.484+0000 7fcc56025640 20 client.4342.objecter shutdown clearing up homeless session... 2024-03-25T19:16:46.485 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:46.484+0000 7fcc56025640 10 client.4342.objecter successfully canceled tick 2024-03-25T19:16:46.485 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:46.485+0000 7fcc56025640 1 -- 172.21.15.6:0/3098579865 >> v1:172.21.15.6:6800/2936652650 conn(0x7fcc3c06e0f0 legacy=0x7fcc3c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:46.485 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:46.485+0000 7fcc56025640 10 monclient: shutdown 2024-03-25T19:16:46.486 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:46.485+0000 7fcc56025640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:16:46.486 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:46.485+0000 7fcc56025640 1 -- 172.21.15.6:0/3098579865 >> v1:172.21.15.192:6789/0 conn(0x7fcc500b44b0 legacy=0x7fcc501df5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:46.486 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:46.485+0000 7fcc56025640 1 -- 172.21.15.6:0/3098579865 shutdown_connections 2024-03-25T19:16:46.486 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:46.485+0000 7fcc56025640 1 -- 172.21.15.6:0/3098579865 >> 172.21.15.6:0/3098579865 conn(0x7fcc500617a0 msgr2=0x7fcc500b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:16:46.486 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:46.485+0000 7fcc56025640 1 -- 172.21.15.6:0/3098579865 shutdown_connections 2024-03-25T19:16:46.486 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:46.485+0000 7fcc56025640 1 -- 172.21.15.6:0/3098579865 wait complete. 2024-03-25T19:16:46.486 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:46.485+0000 7fcc56025640 1 librados: shutdown 2024-03-25T19:16:46.497 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:16:53.302 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:16:53.422 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.422+0000 7fd9624c1640 10 monclient: get_monmap_and_config 2024-03-25T19:16:53.423 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.422+0000 7fd9624c1640 10 monclient: build_initial_monmap 2024-03-25T19:16:53.423 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.422+0000 7fd9624c1640 10 monclient: monmap: 2024-03-25T19:16:53.423 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:16:53.423 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:53.423 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:16:53.423899+0000 2024-03-25T19:16:53.423 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:16:53.423899+0000 2024-03-25T19:16:53.423 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:16:53.423 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:53.423 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:16:53.423 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:16:53.423 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:16:53.423 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:53.425 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.423+0000 7fd9624c1640 1 Processor -- start 2024-03-25T19:16:53.425 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.423+0000 7fd9624c1640 1 -- start start 2024-03-25T19:16:53.425 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.423+0000 7fd9624c1640 10 monclient: init 2024-03-25T19:16:53.425 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.424+0000 7fd9624c1640 10 monclient: _reopen_session rank -1 2024-03-25T19:16:53.425 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.424+0000 7fd9624c1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:16:53.425 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.424+0000 7fd9624c1640 10 monclient(hunting): picked mon.noname-a con 0x7fd95c0b4ff0 addr v1:172.21.15.6:6789/0 2024-03-25T19:16:53.425 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.424+0000 7fd9624c1640 10 monclient(hunting): picked mon.noname-c con 0x7fd95c0b4670 addr v1:172.21.15.192:6789/0 2024-03-25T19:16:53.425 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.424+0000 7fd9624c1640 10 monclient(hunting): picked mon.noname-b con 0x7fd95c13c480 addr v1:172.21.15.6:6790/0 2024-03-25T19:16:53.425 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.424+0000 7fd9624c1640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd95c0ab0e0 con 0x7fd95c0b4ff0 2024-03-25T19:16:53.425 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.424+0000 7fd9624c1640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd95c0ab3c0 con 0x7fd95c0b4670 2024-03-25T19:16:53.426 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.424+0000 7fd9624c1640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd95c0a9f30 con 0x7fd95c13c480 2024-03-25T19:16:53.426 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.424+0000 7fd9624c1640 10 monclient(hunting): _renew_subs 2024-03-25T19:16:53.426 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.424+0000 7fd9624c1640 10 monclient(hunting): authenticate will time out at 6117.664494s 2024-03-25T19:16:53.432 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.432+0000 7fd95b7fe640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fd95c0b4670 0x7fd95c1425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:39526/0 (socket says 172.21.15.6:39526) 2024-03-25T19:16:53.432 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.432+0000 7fd95b7fe640 1 -- 172.21.15.6:0/169103116 learned_addr learned my addr 172.21.15.6:0/169103116 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:16:53.452 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.452+0000 7fd95affd640 1 -- 172.21.15.6:0/169103116 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4130459674 0 0) 0x7fd95c0a9f30 con 0x7fd95c13c480 2024-03-25T19:16:53.452 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.453+0000 7fd95affd640 10 monclient(hunting): _negotiate 2024-03-25T19:16:53.453 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.453+0000 7fd95affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:16:53.453 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.453+0000 7fd95affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:16:53.453 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.453+0000 7fd95affd640 10 monclient(hunting): my global_id is 4343 2024-03-25T19:16:53.453 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.453+0000 7fd95affd640 1 -- 172.21.15.6:0/169103116 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd944003520 con 0x7fd95c13c480 2024-03-25T19:16:53.457 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.457+0000 7fd95affd640 1 -- 172.21.15.6:0/169103116 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3870914295 0 0) 0x7fd95c0ab3c0 con 0x7fd95c0b4670 2024-03-25T19:16:53.457 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.457+0000 7fd95affd640 10 monclient(hunting): _negotiate 2024-03-25T19:16:53.457 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.457+0000 7fd95affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:16:53.457 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.457+0000 7fd95affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:16:53.457 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.457+0000 7fd95affd640 10 monclient(hunting): my global_id is 4345 2024-03-25T19:16:53.457 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.457+0000 7fd95affd640 1 -- 172.21.15.6:0/169103116 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd95c0a9f30 con 0x7fd95c0b4670 2024-03-25T19:16:53.458 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.458+0000 7fd95affd640 1 -- 172.21.15.6:0/169103116 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 135582276 0 0) 0x7fd95c0ab0e0 con 0x7fd95c0b4ff0 2024-03-25T19:16:53.458 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.458+0000 7fd95affd640 10 monclient(hunting): _negotiate 2024-03-25T19:16:53.458 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.458+0000 7fd95affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:16:53.458 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.458+0000 7fd95affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:16:53.458 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.458+0000 7fd95affd640 10 monclient(hunting): my global_id is 4341 2024-03-25T19:16:53.458 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.458+0000 7fd95affd640 1 -- 172.21.15.6:0/169103116 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd95c0ab3c0 con 0x7fd95c0b4ff0 2024-03-25T19:16:53.470 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.469+0000 7fd95affd640 1 -- 172.21.15.6:0/169103116 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3910761567 0 0) 0x7fd944003520 con 0x7fd95c13c480 2024-03-25T19:16:53.470 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.469+0000 7fd95affd640 1 -- 172.21.15.6:0/169103116 >> v1:172.21.15.192:6789/0 conn(0x7fd95c0b4670 legacy=0x7fd95c1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:53.470 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.469+0000 7fd95affd640 1 -- 172.21.15.6:0/169103116 >> v1:172.21.15.6:6789/0 conn(0x7fd95c0b4ff0 legacy=0x7fd95c13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:53.470 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.469+0000 7fd95affd640 10 monclient: _finish_hunting 0 2024-03-25T19:16:53.470 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.469+0000 7fd95affd640 1 monclient: found mon.noname-b 2024-03-25T19:16:53.470 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.469+0000 7fd95affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:16:53.470 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.469+0000 7fd95affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-03-25T19:16:53.470 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.469+0000 7fd95affd640 1 -- 172.21.15.6:0/169103116 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd95c0a9b70 con 0x7fd95c13c480 2024-03-25T19:16:53.470 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.469+0000 7fd95affd640 10 monclient: _finish_auth 0 2024-03-25T19:16:53.470 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.469+0000 7fd95affd640 10 monclient: _check_auth_tickets 2024-03-25T19:16:53.470 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.469+0000 7fd95affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:16:53.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.469+0000 7fd95affd640 1 -- 172.21.15.6:0/169103116 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fd954001a10 con 0x7fd95c13c480 2024-03-25T19:16:53.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.469+0000 7fd9624c1640 5 monclient: authenticate success, global_id 4343 2024-03-25T19:16:53.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.469+0000 7fd9624c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:16:53.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.469+0000 7fd95affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:16:53.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.469+0000 7fd95affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:16:53.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.469+0000 7fd95affd640 10 monclient: dump: 2024-03-25T19:16:53.471 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:16:53.471 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:53.471 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:53.471 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:53.471 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:16:53.471 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:53.471 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:16:53.471 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:16:53.471 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:16:53.472 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:53.472 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.469+0000 7fd9624c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:16:53.494 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.494+0000 7fd95affd640 1 -- 172.21.15.6:0/169103116 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd954001b70 con 0x7fd95c13c480 2024-03-25T19:16:53.494 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.494+0000 7fd95affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:16:53.494 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.494+0000 7fd95affd640 1 -- 172.21.15.6:0/169103116 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fd954001f80 con 0x7fd95c13c480 2024-03-25T19:16:53.494 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.494+0000 7fd9624c1640 10 monclient: get_monmap_and_config success 2024-03-25T19:16:53.495 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.494+0000 7fd9624c1640 4 set_mon_vals no callback set 2024-03-25T19:16:53.495 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.494+0000 7fd95affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:16:53.495 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.494+0000 7fd95affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:16:53.495 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.494+0000 7fd95affd640 10 monclient: dump: 2024-03-25T19:16:53.495 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:16:53.495 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:53.495 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:53.495 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:53.495 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:16:53.495 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:53.495 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:16:53.495 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:16:53.495 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:16:53.495 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:53.498 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.495+0000 7fd9624c1640 10 monclient: shutdown 2024-03-25T19:16:53.498 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.495+0000 7fd9624c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:16:53.498 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.495+0000 7fd9624c1640 1 -- 172.21.15.6:0/169103116 >> v1:172.21.15.6:6790/0 conn(0x7fd95c13c480 legacy=0x7fd95c14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:53.498 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.495+0000 7fd9624c1640 1 -- 172.21.15.6:0/169103116 shutdown_connections 2024-03-25T19:16:53.498 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.495+0000 7fd9624c1640 1 -- 172.21.15.6:0/169103116 >> 172.21.15.6:0/169103116 conn(0x7fd95c0617a0 msgr2=0x7fd95c061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:16:53.498 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.495+0000 7fd9624c1640 1 -- 172.21.15.6:0/169103116 shutdown_connections 2024-03-25T19:16:53.498 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.495+0000 7fd9624c1640 1 -- 172.21.15.6:0/169103116 wait complete. 2024-03-25T19:16:53.498 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.496+0000 7fd9624c1640 10 monclient: build_initial_monmap 2024-03-25T19:16:53.498 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.496+0000 7fd9624c1640 10 monclient: monmap: 2024-03-25T19:16:53.498 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:16:53.498 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:16:53.498 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:16:53.499 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:16:53.499 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:16:53.499 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:16:53.499 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:16:53.499 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:16:53.499 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:16:53.499 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:53.499 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.496+0000 7fd9624c1640 1 librados: starting msgr at 2024-03-25T19:16:53.499 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.496+0000 7fd9624c1640 1 librados: starting objecter 2024-03-25T19:16:53.499 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.496+0000 7fd9624c1640 1 Processor -- start 2024-03-25T19:16:53.499 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.496+0000 7fd9624c1640 1 -- start start 2024-03-25T19:16:53.499 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.496+0000 7fd9624c1640 1 librados: setting wanted keys 2024-03-25T19:16:53.499 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.496+0000 7fd9624c1640 1 librados: calling monclient init 2024-03-25T19:16:53.499 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.496+0000 7fd9624c1640 10 monclient: init 2024-03-25T19:16:53.500 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.497+0000 7fd9624c1640 10 monclient: _reopen_session rank -1 2024-03-25T19:16:53.500 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.497+0000 7fd9624c1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:16:53.500 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.497+0000 7fd9624c1640 10 monclient(hunting): picked mon.noname-a con 0x7fd95c0b4670 addr v1:172.21.15.6:6789/0 2024-03-25T19:16:53.500 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.497+0000 7fd9624c1640 10 monclient(hunting): picked mon.noname-c con 0x7fd95c0b4ff0 addr v1:172.21.15.6:6790/0 2024-03-25T19:16:53.500 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.497+0000 7fd9624c1640 10 monclient(hunting): picked mon.noname-b con 0x7fd95c13c480 addr v1:172.21.15.192:6789/0 2024-03-25T19:16:53.500 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.497+0000 7fd9624c1640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd95c135d10 con 0x7fd95c0b4670 2024-03-25T19:16:53.500 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.497+0000 7fd9624c1640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd95c0ac080 con 0x7fd95c13c480 2024-03-25T19:16:53.500 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.497+0000 7fd9624c1640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd95c0ac360 con 0x7fd95c0b4ff0 2024-03-25T19:16:53.500 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.497+0000 7fd9624c1640 10 monclient(hunting): _renew_subs 2024-03-25T19:16:53.500 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.497+0000 7fd9624c1640 10 monclient(hunting): authenticate will time out at 6117.737193s 2024-03-25T19:16:53.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.506+0000 7fd960a5e640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fd95c13c480 0x7fd95c1e26e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:39586/0 (socket says 172.21.15.6:39586) 2024-03-25T19:16:53.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.506+0000 7fd960a5e640 1 -- 172.21.15.6:0/1334526268 learned_addr learned my addr 172.21.15.6:0/1334526268 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:16:53.507 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.507+0000 7fd958ff9640 10 client.?.objecter ms_handle_connect 0x7fd95c13c480 2024-03-25T19:16:53.507 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.507+0000 7fd958ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:16:53.514 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.515+0000 7fd958ff9640 10 client.?.objecter ms_handle_connect 0x7fd95c0b4670 2024-03-25T19:16:53.514 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.515+0000 7fd958ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:16:53.518 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.519+0000 7fd958ff9640 10 client.?.objecter ms_handle_connect 0x7fd95c0b4ff0 2024-03-25T19:16:53.519 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.519+0000 7fd958ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:16:53.520 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.520+0000 7fd958ff9640 1 -- 172.21.15.6:0/1334526268 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1232179834 0 0) 0x7fd95c0ac080 con 0x7fd95c13c480 2024-03-25T19:16:53.520 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.520+0000 7fd958ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:16:53.520 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.520+0000 7fd958ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:16:53.520 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.520+0000 7fd958ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:16:53.520 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.520+0000 7fd958ff9640 10 monclient(hunting): my global_id is 4348 2024-03-25T19:16:53.520 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.520+0000 7fd958ff9640 1 -- 172.21.15.6:0/1334526268 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd948004eb0 con 0x7fd95c13c480 2024-03-25T19:16:53.525 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.525+0000 7fd958ff9640 1 -- 172.21.15.6:0/1334526268 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3825656004 0 0) 0x7fd95c0ac360 con 0x7fd95c0b4ff0 2024-03-25T19:16:53.525 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.525+0000 7fd958ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:16:53.525 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.525+0000 7fd958ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:16:53.526 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.525+0000 7fd958ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:16:53.526 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.525+0000 7fd958ff9640 10 monclient(hunting): my global_id is 4346 2024-03-25T19:16:53.526 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.525+0000 7fd958ff9640 1 -- 172.21.15.6:0/1334526268 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd95c0ac080 con 0x7fd95c0b4ff0 2024-03-25T19:16:53.527 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.527+0000 7fd958ff9640 1 -- 172.21.15.6:0/1334526268 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1901877031 0 0) 0x7fd95c135d10 con 0x7fd95c0b4670 2024-03-25T19:16:53.527 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.527+0000 7fd958ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:16:53.527 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.527+0000 7fd958ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:16:53.527 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.527+0000 7fd958ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:16:53.527 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.527+0000 7fd958ff9640 10 monclient(hunting): my global_id is 4344 2024-03-25T19:16:53.527 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.527+0000 7fd958ff9640 1 -- 172.21.15.6:0/1334526268 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd95c0ac360 con 0x7fd95c0b4670 2024-03-25T19:16:53.535 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.535+0000 7fd958ff9640 1 -- 172.21.15.6:0/1334526268 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1187388698 0 0) 0x7fd948004eb0 con 0x7fd95c13c480 2024-03-25T19:16:53.535 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.535+0000 7fd958ff9640 1 -- 172.21.15.6:0/1334526268 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd95c135d10 con 0x7fd95c13c480 2024-03-25T19:16:53.535 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.535+0000 7fd958ff9640 1 -- 172.21.15.6:0/1334526268 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fd954001ec0 con 0x7fd95c13c480 2024-03-25T19:16:53.535 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.535+0000 7fd958ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:16:53.535 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.535+0000 7fd958ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:16:53.536 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.535+0000 7fd958ff9640 10 monclient(hunting): dump: 2024-03-25T19:16:53.536 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:16:53.536 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:53.536 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:53.536 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:53.536 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:16:53.536 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:53.536 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:16:53.536 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:16:53.536 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:16:53.536 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:53.536 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.535+0000 7fd958ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:16:53.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.545+0000 7fd958ff9640 1 -- 172.21.15.6:0/1334526268 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2334648806 0 0) 0x7fd95c0ac360 con 0x7fd95c0b4670 2024-03-25T19:16:53.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.545+0000 7fd958ff9640 1 -- 172.21.15.6:0/1334526268 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd948004eb0 con 0x7fd95c0b4670 2024-03-25T19:16:53.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.545+0000 7fd958ff9640 1 -- 172.21.15.6:0/1334526268 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fd94c001830 con 0x7fd95c0b4670 2024-03-25T19:16:53.546 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.545+0000 7fd958ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:16:53.546 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.545+0000 7fd958ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:16:53.546 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.545+0000 7fd958ff9640 10 monclient(hunting): dump: 2024-03-25T19:16:53.546 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:16:53.546 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:53.546 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:53.546 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:53.546 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:16:53.546 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:53.546 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:16:53.546 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:16:53.546 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:16:53.546 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:53.549 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.549+0000 7fd958ff9640 1 -- 172.21.15.6:0/1334526268 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 176058181 0 0) 0x7fd95c0ac080 con 0x7fd95c0b4ff0 2024-03-25T19:16:53.549 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.549+0000 7fd958ff9640 1 -- 172.21.15.6:0/1334526268 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd95c0ac360 con 0x7fd95c0b4ff0 2024-03-25T19:16:53.549 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.549+0000 7fd958ff9640 1 -- 172.21.15.6:0/1334526268 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fd950001970 con 0x7fd95c0b4ff0 2024-03-25T19:16:53.549 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.549+0000 7fd958ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:16:53.549 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.549+0000 7fd958ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:16:53.550 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.549+0000 7fd958ff9640 10 monclient(hunting): dump: 2024-03-25T19:16:53.550 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:16:53.550 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:53.550 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:53.550 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:53.550 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:16:53.550 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:53.550 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:16:53.550 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:16:53.550 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:16:53.550 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:53.558 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.558+0000 7fd958ff9640 1 -- 172.21.15.6:0/1334526268 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 488431118 0 0) 0x7fd95c135d10 con 0x7fd95c13c480 2024-03-25T19:16:53.558 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.558+0000 7fd958ff9640 1 -- 172.21.15.6:0/1334526268 >> v1:172.21.15.6:6790/0 conn(0x7fd95c0b4ff0 legacy=0x7fd95c136890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:53.558 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.558+0000 7fd958ff9640 1 -- 172.21.15.6:0/1334526268 >> v1:172.21.15.6:6789/0 conn(0x7fd95c0b4670 legacy=0x7fd95c136180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:53.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.558+0000 7fd958ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:16:53.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.558+0000 7fd958ff9640 1 monclient: found mon.b 2024-03-25T19:16:53.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.558+0000 7fd958ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:16:53.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.558+0000 7fd958ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:16:53.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.558+0000 7fd958ff9640 1 -- 172.21.15.6:0/1334526268 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd95c0ac640 con 0x7fd95c13c480 2024-03-25T19:16:53.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.558+0000 7fd958ff9640 10 monclient: _finish_auth 0 2024-03-25T19:16:53.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.558+0000 7fd958ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:16:53.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.558+0000 7fd958ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:16:53.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.558+0000 7fd9624c1640 5 monclient: authenticate success, global_id 4348 2024-03-25T19:16:53.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.558+0000 7fd9624c1640 10 monclient: _renew_subs 2024-03-25T19:16:53.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.558+0000 7fd9624c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:16:53.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.558+0000 7fd9624c1640 1 -- 172.21.15.6:0/1334526268 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd95c13b060 con 0x7fd95c13c480 2024-03-25T19:16:53.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.558+0000 7fd9624c1640 10 client.4348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:16:53.560 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.558+0000 7fd9624c1640 10 monclient: _renew_subs 2024-03-25T19:16:53.560 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.558+0000 7fd9624c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:16:53.560 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.558+0000 7fd9624c1640 1 -- 172.21.15.6:0/1334526268 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd95c13ad60 con 0x7fd95c13c480 2024-03-25T19:16:53.560 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.558+0000 7fd9624c1640 1 librados: init done 2024-03-25T19:16:53.560 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.559+0000 7fd9624c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:16:53.561 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.559+0000 7fd9624c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:16:53.561 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.559+0000 7fd9624c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:16:53.561 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.559+0000 7fd9624c1640 1 -- 172.21.15.6:0/1334526268 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd95c06dbd0 con 0x7fd95c13c480 2024-03-25T19:16:53.836 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.836+0000 7fd958ff9640 1 -- 172.21.15.6:0/1334526268 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd954003c20 con 0x7fd95c13c480 2024-03-25T19:16:53.836 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.836+0000 7fd958ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:16:53.836 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.836+0000 7fd958ff9640 1 -- 172.21.15.6:0/1334526268 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fd954002100 con 0x7fd95c13c480 2024-03-25T19:16:53.836 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.836+0000 7fd958ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:16:53.836 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.836+0000 7fd958ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:16:53.836 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.836+0000 7fd958ff9640 10 monclient: dump: 2024-03-25T19:16:53.836 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:16:53.836 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:16:53.837 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:53.837 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:16:53.837 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:16:53.837 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:16:53.837 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:16:53.837 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:16:53.837 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:16:53.837 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:16:53.837 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.836+0000 7fd95a7fc640 4 set_mon_vals no callback set 2024-03-25T19:16:53.838 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.837+0000 7fd958ff9640 1 -- 172.21.15.6:0/1334526268 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7fd9540040f0 con 0x7fd95c13c480 2024-03-25T19:16:53.838 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.837+0000 7fd95bfff640 10 monclient: get_auth_request con 0x7fd94806e0f0 auth_method 0 2024-03-25T19:16:53.838 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.838+0000 7fd958ff9640 10 client.4348.objecter ms_handle_connect 0x7fd94806e0f0 2024-03-25T19:16:53.843 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.843+0000 7fd958ff9640 1 -- 172.21.15.6:0/1334526268 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7fd954054ad0 con 0x7fd95c13c480 2024-03-25T19:16:53.843 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.843+0000 7fd958ff9640 10 client.4348.objecter ms_dispatch 0x7fd95c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:16:53.843 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.843+0000 7fd958ff9640 3 client.4348.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:16:53.843 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.843+0000 7fd958ff9640 3 client.4348.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:16:53.843 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.843+0000 7fd958ff9640 20 client.4348.objecter dump_active .. 0 homeless 2024-03-25T19:16:53.844 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.843+0000 7fd958ff9640 1 -- 172.21.15.6:0/1334526268 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd9540505e0 con 0x7fd95c13c480 2024-03-25T19:16:53.844 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.844+0000 7fd958ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:16:53.844 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:53.844+0000 7fd958ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:16:54.089 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:54.088+0000 7fd9624c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:16:54.089 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:54.088+0000 7fd9624c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:16:54.089 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:54.088+0000 7fd9624c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:16:54.089 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:54.088+0000 7fd9624c1640 1 -- 172.21.15.6:0/1334526268 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd95c143410 con 0x7fd95c13c480 2024-03-25T19:16:54.168 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:54.168+0000 7fd958ff9640 1 -- 172.21.15.6:0/1334526268 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7fd954052510 con 0x7fd95c13c480 2024-03-25T19:16:54.169 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:54.168+0000 7fd958ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:16:54.169 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:54.168+0000 7fd958ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:16:54.169 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:16:54.169 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:16:54.171 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:54.171+0000 7fd9624c1640 10 librados: watch_flush enter 2024-03-25T19:16:54.171 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:54.171+0000 7fd9624c1640 10 librados: watch_flush exit 2024-03-25T19:16:54.171 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:54.171+0000 7fd9624c1640 20 client.4348.objecter shutdown clearing up homeless session... 2024-03-25T19:16:54.172 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:54.171+0000 7fd9624c1640 10 client.4348.objecter successfully canceled tick 2024-03-25T19:16:54.172 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:54.171+0000 7fd9624c1640 1 -- 172.21.15.6:0/1334526268 >> v1:172.21.15.6:6800/2936652650 conn(0x7fd94806e0f0 legacy=0x7fd9480705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:54.172 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:54.171+0000 7fd9624c1640 10 monclient: shutdown 2024-03-25T19:16:54.172 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:54.171+0000 7fd9624c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:16:54.172 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:54.171+0000 7fd9624c1640 1 -- 172.21.15.6:0/1334526268 >> v1:172.21.15.192:6789/0 conn(0x7fd95c13c480 legacy=0x7fd95c1e26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:16:54.172 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:54.171+0000 7fd9624c1640 1 -- 172.21.15.6:0/1334526268 shutdown_connections 2024-03-25T19:16:54.172 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:54.171+0000 7fd9624c1640 1 -- 172.21.15.6:0/1334526268 >> 172.21.15.6:0/1334526268 conn(0x7fd95c0617a0 msgr2=0x7fd95c140760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:16:54.172 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:54.171+0000 7fd9624c1640 1 -- 172.21.15.6:0/1334526268 shutdown_connections 2024-03-25T19:16:54.172 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:54.171+0000 7fd9624c1640 1 -- 172.21.15.6:0/1334526268 wait complete. 2024-03-25T19:16:54.172 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:16:54.171+0000 7fd9624c1640 1 librados: shutdown 2024-03-25T19:16:54.183 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:17:00.990 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:17:01.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.104+0000 7f7760233640 10 monclient: get_monmap_and_config 2024-03-25T19:17:01.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.104+0000 7f7760233640 10 monclient: build_initial_monmap 2024-03-25T19:17:01.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.104+0000 7f7760233640 10 monclient: monmap: 2024-03-25T19:17:01.104 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:17:01.104 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:01.104 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:17:01.105352+0000 2024-03-25T19:17:01.104 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:17:01.105352+0000 2024-03-25T19:17:01.105 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:17:01.105 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:01.105 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:17:01.105 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:17:01.105 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:17:01.105 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:01.106 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.105+0000 7f7760233640 1 Processor -- start 2024-03-25T19:17:01.106 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.105+0000 7f7760233640 1 -- start start 2024-03-25T19:17:01.106 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.105+0000 7f7760233640 10 monclient: init 2024-03-25T19:17:01.107 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.105+0000 7f7760233640 10 monclient: _reopen_session rank -1 2024-03-25T19:17:01.107 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.105+0000 7f7760233640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:17:01.107 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.105+0000 7f7760233640 10 monclient(hunting): picked mon.noname-b con 0x7f77580b4e70 addr v1:172.21.15.6:6790/0 2024-03-25T19:17:01.107 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.105+0000 7f7760233640 10 monclient(hunting): picked mon.noname-a con 0x7f77580b44b0 addr v1:172.21.15.6:6789/0 2024-03-25T19:17:01.107 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.106+0000 7f7760233640 10 monclient(hunting): picked mon.noname-c con 0x7f77580b1b00 addr v1:172.21.15.192:6789/0 2024-03-25T19:17:01.107 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.106+0000 7f7760233640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77580a96f0 con 0x7f77580b44b0 2024-03-25T19:17:01.107 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.106+0000 7f7760233640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77580ab0e0 con 0x7f77580b1b00 2024-03-25T19:17:01.107 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.106+0000 7f7760233640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7758145f70 con 0x7f77580b4e70 2024-03-25T19:17:01.107 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.106+0000 7f7760233640 10 monclient(hunting): _renew_subs 2024-03-25T19:17:01.107 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.106+0000 7f7760233640 10 monclient(hunting): authenticate will time out at 6125.345852s 2024-03-25T19:17:01.110 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.110+0000 7f775e7d0640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f77580b1b00 0x7f775814dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:35170/0 (socket says 172.21.15.6:35170) 2024-03-25T19:17:01.110 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.110+0000 7f775e7d0640 1 -- 172.21.15.6:0/1026878986 learned_addr learned my addr 172.21.15.6:0/1026878986 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:17:01.127 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.127+0000 7f775cfcd640 1 -- 172.21.15.6:0/1026878986 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 307331000 0 0) 0x7f77580a96f0 con 0x7f77580b44b0 2024-03-25T19:17:01.127 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.127+0000 7f775cfcd640 10 monclient(hunting): _negotiate 2024-03-25T19:17:01.127 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.127+0000 7f775cfcd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:01.127 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.127+0000 7f775cfcd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:01.127 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.127+0000 7f775cfcd640 10 monclient(hunting): my global_id is 4347 2024-03-25T19:17:01.127 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.127+0000 7f775cfcd640 1 -- 172.21.15.6:0/1026878986 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7744003520 con 0x7f77580b44b0 2024-03-25T19:17:01.130 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.130+0000 7f775cfcd640 1 -- 172.21.15.6:0/1026878986 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1823748639 0 0) 0x7f7758145f70 con 0x7f77580b4e70 2024-03-25T19:17:01.130 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.130+0000 7f775cfcd640 10 monclient(hunting): _negotiate 2024-03-25T19:17:01.130 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.130+0000 7f775cfcd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:01.130 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.130+0000 7f775cfcd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:01.130 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.130+0000 7f775cfcd640 10 monclient(hunting): my global_id is 4349 2024-03-25T19:17:01.130 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.130+0000 7f775cfcd640 1 -- 172.21.15.6:0/1026878986 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77580a96f0 con 0x7f77580b4e70 2024-03-25T19:17:01.137 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.137+0000 7f775cfcd640 1 -- 172.21.15.6:0/1026878986 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4259303570 0 0) 0x7f7744003520 con 0x7f77580b44b0 2024-03-25T19:17:01.137 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.137+0000 7f775cfcd640 1 -- 172.21.15.6:0/1026878986 >> v1:172.21.15.6:6790/0 conn(0x7f77580b4e70 legacy=0x7f775813f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:01.137 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.137+0000 7f775cfcd640 1 -- 172.21.15.6:0/1026878986 >> v1:172.21.15.192:6789/0 conn(0x7f77580b1b00 legacy=0x7f775814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:01.137 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.137+0000 7f775cfcd640 10 monclient: _finish_hunting 0 2024-03-25T19:17:01.137 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.137+0000 7f775cfcd640 1 monclient: found mon.noname-a 2024-03-25T19:17:01.137 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.137+0000 7f775cfcd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:17:01.138 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.137+0000 7f775cfcd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-03-25T19:17:01.138 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.137+0000 7f775cfcd640 1 -- 172.21.15.6:0/1026878986 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77580a9b80 con 0x7f77580b44b0 2024-03-25T19:17:01.138 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.137+0000 7f775cfcd640 10 monclient: _finish_auth 0 2024-03-25T19:17:01.138 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.137+0000 7f775cfcd640 10 monclient: _check_auth_tickets 2024-03-25T19:17:01.138 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.137+0000 7f775cfcd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:17:01.138 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.137+0000 7f775cfcd640 1 -- 172.21.15.6:0/1026878986 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f7748001690 con 0x7f77580b44b0 2024-03-25T19:17:01.138 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.137+0000 7f775cfcd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:17:01.138 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.137+0000 7f775cfcd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:17:01.138 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.137+0000 7f775cfcd640 10 monclient: dump: 2024-03-25T19:17:01.138 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:17:01.138 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:01.138 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:01.138 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:01.139 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:17:01.139 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:01.139 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:17:01.139 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:17:01.139 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:17:01.139 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:01.139 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.137+0000 7f7760233640 5 monclient: authenticate success, global_id 4347 2024-03-25T19:17:01.139 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.137+0000 7f7760233640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:17:01.159 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.159+0000 7f775cfcd640 1 -- 172.21.15.6:0/1026878986 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77480017f0 con 0x7f77580b44b0 2024-03-25T19:17:01.159 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.159+0000 7f775cfcd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:17:01.159 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.159+0000 7f775cfcd640 1 -- 172.21.15.6:0/1026878986 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f7748001c00 con 0x7f77580b44b0 2024-03-25T19:17:01.159 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.159+0000 7f7760233640 10 monclient: get_monmap_and_config success 2024-03-25T19:17:01.159 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.159+0000 7f7760233640 4 set_mon_vals no callback set 2024-03-25T19:17:01.160 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.159+0000 7f775cfcd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:17:01.160 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.159+0000 7f775cfcd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:17:01.160 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.159+0000 7f775cfcd640 10 monclient: dump: 2024-03-25T19:17:01.160 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:17:01.160 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:01.160 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:01.160 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:01.160 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:17:01.160 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:01.160 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:17:01.160 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:17:01.160 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:17:01.160 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:01.162 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.160+0000 7f7760233640 10 monclient: shutdown 2024-03-25T19:17:01.163 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.160+0000 7f7760233640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:17:01.163 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.160+0000 7f7760233640 1 -- 172.21.15.6:0/1026878986 >> v1:172.21.15.6:6789/0 conn(0x7f77580b44b0 legacy=0x7f7758145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:01.163 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.160+0000 7f7760233640 1 -- 172.21.15.6:0/1026878986 shutdown_connections 2024-03-25T19:17:01.163 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.160+0000 7f7760233640 1 -- 172.21.15.6:0/1026878986 >> 172.21.15.6:0/1026878986 conn(0x7f77580617a0 msgr2=0x7f7758061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:17:01.163 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.160+0000 7f7760233640 1 -- 172.21.15.6:0/1026878986 shutdown_connections 2024-03-25T19:17:01.163 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.160+0000 7f7760233640 1 -- 172.21.15.6:0/1026878986 wait complete. 2024-03-25T19:17:01.163 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.161+0000 7f7760233640 10 monclient: build_initial_monmap 2024-03-25T19:17:01.163 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.161+0000 7f7760233640 10 monclient: monmap: 2024-03-25T19:17:01.163 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:17:01.163 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:17:01.163 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:17:01.163 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:17:01.163 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:17:01.163 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:17:01.164 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:17:01.164 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:17:01.164 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:17:01.164 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:01.164 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.161+0000 7f7760233640 1 librados: starting msgr at 2024-03-25T19:17:01.164 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.161+0000 7f7760233640 1 librados: starting objecter 2024-03-25T19:17:01.164 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.161+0000 7f7760233640 1 Processor -- start 2024-03-25T19:17:01.164 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.161+0000 7f7760233640 1 -- start start 2024-03-25T19:17:01.164 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.161+0000 7f7760233640 1 librados: setting wanted keys 2024-03-25T19:17:01.164 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.161+0000 7f7760233640 1 librados: calling monclient init 2024-03-25T19:17:01.164 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.161+0000 7f7760233640 10 monclient: init 2024-03-25T19:17:01.164 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.162+0000 7f7760233640 10 monclient: _reopen_session rank -1 2024-03-25T19:17:01.164 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.162+0000 7f7760233640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:17:01.164 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.162+0000 7f7760233640 10 monclient(hunting): picked mon.noname-b con 0x7f77580b1b00 addr v1:172.21.15.192:6789/0 2024-03-25T19:17:01.165 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.162+0000 7f7760233640 10 monclient(hunting): picked mon.noname-a con 0x7f77580b44b0 addr v1:172.21.15.6:6789/0 2024-03-25T19:17:01.165 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.162+0000 7f7760233640 10 monclient(hunting): picked mon.noname-c con 0x7f77580b4e70 addr v1:172.21.15.6:6790/0 2024-03-25T19:17:01.165 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.162+0000 7f7760233640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77581db8b0 con 0x7f77580b44b0 2024-03-25T19:17:01.165 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.162+0000 7f7760233640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7758142050 con 0x7f77580b1b00 2024-03-25T19:17:01.165 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.162+0000 7f7760233640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77581422f0 con 0x7f77580b4e70 2024-03-25T19:17:01.165 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.162+0000 7f7760233640 10 monclient(hunting): _renew_subs 2024-03-25T19:17:01.165 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.162+0000 7f7760233640 10 monclient(hunting): authenticate will time out at 6125.402245s 2024-03-25T19:17:01.176 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.176+0000 7f775d7ce640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f77580b44b0 0x7f77581402d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50418/0 (socket says 172.21.15.6:50418) 2024-03-25T19:17:01.176 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.176+0000 7f775d7ce640 1 -- 172.21.15.6:0/2730291643 learned_addr learned my addr 172.21.15.6:0/2730291643 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:17:01.178 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.178+0000 7f774e7fc640 10 client.?.objecter ms_handle_connect 0x7f77580b44b0 2024-03-25T19:17:01.178 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.178+0000 7f774e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:17:01.183 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.183+0000 7f774e7fc640 10 client.?.objecter ms_handle_connect 0x7f77580b1b00 2024-03-25T19:17:01.183 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.183+0000 7f774e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:17:01.194 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.194+0000 7f774e7fc640 10 client.?.objecter ms_handle_connect 0x7f77580b4e70 2024-03-25T19:17:01.194 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.194+0000 7f774e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:17:01.197 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.197+0000 7f774e7fc640 1 -- 172.21.15.6:0/2730291643 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1528670740 0 0) 0x7f77581db8b0 con 0x7f77580b44b0 2024-03-25T19:17:01.197 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.197+0000 7f774e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:17:01.197 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.197+0000 7f774e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:01.197 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.197+0000 7f774e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:01.197 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.197+0000 7f774e7fc640 10 monclient(hunting): my global_id is 4350 2024-03-25T19:17:01.197 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.197+0000 7f774e7fc640 1 -- 172.21.15.6:0/2730291643 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f773c004eb0 con 0x7f77580b44b0 2024-03-25T19:17:01.206 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.206+0000 7f774e7fc640 1 -- 172.21.15.6:0/2730291643 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2609369439 0 0) 0x7f77581422f0 con 0x7f77580b4e70 2024-03-25T19:17:01.206 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.206+0000 7f774e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:17:01.206 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.206+0000 7f774e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:01.206 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.206+0000 7f774e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:01.206 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.206+0000 7f774e7fc640 10 monclient(hunting): my global_id is 4352 2024-03-25T19:17:01.206 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.206+0000 7f774e7fc640 1 -- 172.21.15.6:0/2730291643 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77581db8b0 con 0x7f77580b4e70 2024-03-25T19:17:01.207 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.207+0000 7f774e7fc640 1 -- 172.21.15.6:0/2730291643 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2247680535 0 0) 0x7f7758142050 con 0x7f77580b1b00 2024-03-25T19:17:01.207 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.207+0000 7f774e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:17:01.207 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.207+0000 7f774e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:01.207 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.207+0000 7f774e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:01.207 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.207+0000 7f774e7fc640 10 monclient(hunting): my global_id is 4354 2024-03-25T19:17:01.207 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.207+0000 7f774e7fc640 1 -- 172.21.15.6:0/2730291643 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77581422f0 con 0x7f77580b1b00 2024-03-25T19:17:01.226 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.226+0000 7f774e7fc640 1 -- 172.21.15.6:0/2730291643 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 295020692 0 0) 0x7f77581422f0 con 0x7f77580b1b00 2024-03-25T19:17:01.226 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.226+0000 7f774e7fc640 1 -- 172.21.15.6:0/2730291643 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7758142050 con 0x7f77580b1b00 2024-03-25T19:17:01.227 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.226+0000 7f774e7fc640 1 -- 172.21.15.6:0/2730291643 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f77540017d0 con 0x7f77580b1b00 2024-03-25T19:17:01.227 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.226+0000 7f774e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:17:01.227 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.226+0000 7f774e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:17:01.227 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.226+0000 7f774e7fc640 10 monclient(hunting): dump: 2024-03-25T19:17:01.227 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:17:01.227 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:01.227 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:01.227 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:01.227 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:17:01.227 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:01.227 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:17:01.227 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:17:01.227 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:17:01.227 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:01.228 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.226+0000 7f774e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:17:01.231 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.231+0000 7f774e7fc640 1 -- 172.21.15.6:0/2730291643 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2345611943 0 0) 0x7f77581db8b0 con 0x7f77580b4e70 2024-03-25T19:17:01.231 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.231+0000 7f774e7fc640 1 -- 172.21.15.6:0/2730291643 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77581422f0 con 0x7f77580b4e70 2024-03-25T19:17:01.231 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.231+0000 7f774e7fc640 1 -- 172.21.15.6:0/2730291643 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f7750001ba0 con 0x7f77580b4e70 2024-03-25T19:17:01.231 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.231+0000 7f774e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:17:01.231 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.231+0000 7f774e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:17:01.231 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.231+0000 7f774e7fc640 10 monclient(hunting): dump: 2024-03-25T19:17:01.231 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:17:01.231 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:01.231 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:01.231 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:01.231 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:17:01.231 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:01.232 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:17:01.232 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:17:01.232 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:17:01.232 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:01.239 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.239+0000 7f774e7fc640 1 -- 172.21.15.6:0/2730291643 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2797430411 0 0) 0x7f773c004eb0 con 0x7f77580b44b0 2024-03-25T19:17:01.239 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.239+0000 7f774e7fc640 1 -- 172.21.15.6:0/2730291643 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77581db8b0 con 0x7f77580b44b0 2024-03-25T19:17:01.239 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.239+0000 7f774e7fc640 1 -- 172.21.15.6:0/2730291643 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f7748001690 con 0x7f77580b44b0 2024-03-25T19:17:01.239 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.239+0000 7f774e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:17:01.239 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.239+0000 7f774e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:17:01.239 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.239+0000 7f774e7fc640 10 monclient(hunting): dump: 2024-03-25T19:17:01.239 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:17:01.239 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:01.240 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:01.240 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:01.240 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:17:01.240 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:01.240 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:17:01.240 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:17:01.240 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:17:01.240 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:01.257 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.256+0000 7f774e7fc640 1 -- 172.21.15.6:0/2730291643 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3089152988 0 0) 0x7f7758142050 con 0x7f77580b1b00 2024-03-25T19:17:01.257 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.257+0000 7f774e7fc640 1 -- 172.21.15.6:0/2730291643 >> v1:172.21.15.6:6790/0 conn(0x7f77580b4e70 legacy=0x7f77581409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:01.257 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.257+0000 7f774e7fc640 1 -- 172.21.15.6:0/2730291643 >> v1:172.21.15.6:6789/0 conn(0x7f77580b44b0 legacy=0x7f77581402d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:01.257 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.257+0000 7f774e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:17:01.257 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.257+0000 7f774e7fc640 1 monclient: found mon.b 2024-03-25T19:17:01.257 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.257+0000 7f774e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:17:01.257 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.257+0000 7f774e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:17:01.257 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.257+0000 7f774e7fc640 1 -- 172.21.15.6:0/2730291643 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7758139b60 con 0x7f77580b1b00 2024-03-25T19:17:01.257 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.257+0000 7f774e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:17:01.258 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.257+0000 7f774e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:17:01.258 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.257+0000 7f774e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:17:01.258 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.257+0000 7f7760233640 5 monclient: authenticate success, global_id 4354 2024-03-25T19:17:01.258 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.257+0000 7f7760233640 10 monclient: _renew_subs 2024-03-25T19:17:01.258 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.257+0000 7f7760233640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:17:01.258 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.257+0000 7f7760233640 1 -- 172.21.15.6:0/2730291643 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77581db460 con 0x7f77580b1b00 2024-03-25T19:17:01.258 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.257+0000 7f7760233640 10 client.4354.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:17:01.258 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.257+0000 7f7760233640 10 monclient: _renew_subs 2024-03-25T19:17:01.258 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.257+0000 7f7760233640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:17:01.258 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.257+0000 7f7760233640 1 -- 172.21.15.6:0/2730291643 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f77581e3210 con 0x7f77580b1b00 2024-03-25T19:17:01.258 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.257+0000 7f7760233640 1 librados: init done 2024-03-25T19:17:01.259 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.258+0000 7f7760233640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:17:01.259 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.258+0000 7f7760233640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:17:01.259 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.258+0000 7f7760233640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:17:01.259 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.258+0000 7f7760233640 1 -- 172.21.15.6:0/2730291643 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f775804f730 con 0x7f77580b1b00 2024-03-25T19:17:01.554 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.554+0000 7f774e7fc640 1 -- 172.21.15.6:0/2730291643 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7754001ce0 con 0x7f77580b1b00 2024-03-25T19:17:01.554 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.554+0000 7f774e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:17:01.554 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.554+0000 7f774e7fc640 1 -- 172.21.15.6:0/2730291643 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f7754003790 con 0x7f77580b1b00 2024-03-25T19:17:01.555 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.554+0000 7f774e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:17:01.555 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.554+0000 7f774e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:17:01.555 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.554+0000 7f774e7fc640 10 monclient: dump: 2024-03-25T19:17:01.555 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:17:01.555 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:01.555 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:01.555 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:01.555 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:17:01.555 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:01.555 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:17:01.555 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:17:01.555 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:17:01.556 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:01.556 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.554+0000 7f774ffff640 4 set_mon_vals no callback set 2024-03-25T19:17:01.556 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.555+0000 7f774e7fc640 1 -- 172.21.15.6:0/2730291643 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f77540197a0 con 0x7f77580b1b00 2024-03-25T19:17:01.556 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.556+0000 7f775d7ce640 10 monclient: get_auth_request con 0x7f773c06e010 auth_method 0 2024-03-25T19:17:01.556 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.556+0000 7f774e7fc640 10 client.4354.objecter ms_handle_connect 0x7f773c06e010 2024-03-25T19:17:01.558 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.558+0000 7f774e7fc640 1 -- 172.21.15.6:0/2730291643 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f77540534c0 con 0x7f77580b1b00 2024-03-25T19:17:01.558 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.558+0000 7f774e7fc640 10 client.4354.objecter ms_dispatch 0x7f7758000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:17:01.558 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.558+0000 7f774e7fc640 3 client.4354.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:17:01.558 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.558+0000 7f774e7fc640 3 client.4354.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:17:01.558 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.558+0000 7f774e7fc640 20 client.4354.objecter dump_active .. 0 homeless 2024-03-25T19:17:01.561 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.561+0000 7f774e7fc640 1 -- 172.21.15.6:0/2730291643 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f775404ee00 con 0x7f77580b1b00 2024-03-25T19:17:01.561 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.561+0000 7f774e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:17:01.561 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.561+0000 7f774e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:17:01.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.783+0000 7f7760233640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:17:01.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.783+0000 7f7760233640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:17:01.784 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.783+0000 7f7760233640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:17:01.784 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.783+0000 7f7760233640 1 -- 172.21.15.6:0/2730291643 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f77581db6a0 con 0x7f77580b1b00 2024-03-25T19:17:01.825 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.825+0000 7f774e7fc640 1 -- 172.21.15.6:0/2730291643 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f7754050f00 con 0x7f77580b1b00 2024-03-25T19:17:01.825 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.825+0000 7f774e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:17:01.825 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.825+0000 7f774e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:17:01.826 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:17:01.826 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:17:01.828 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.828+0000 7f7760233640 10 librados: watch_flush enter 2024-03-25T19:17:01.828 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.828+0000 7f7760233640 10 librados: watch_flush exit 2024-03-25T19:17:01.828 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.828+0000 7f7760233640 20 client.4354.objecter shutdown clearing up homeless session... 2024-03-25T19:17:01.828 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.828+0000 7f7760233640 10 client.4354.objecter successfully canceled tick 2024-03-25T19:17:01.829 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.828+0000 7f7760233640 1 -- 172.21.15.6:0/2730291643 >> v1:172.21.15.6:6800/2936652650 conn(0x7f773c06e010 legacy=0x7f773c0704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:01.829 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.828+0000 7f7760233640 10 monclient: shutdown 2024-03-25T19:17:01.829 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.828+0000 7f7760233640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:17:01.829 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.828+0000 7f7760233640 1 -- 172.21.15.6:0/2730291643 >> v1:172.21.15.192:6789/0 conn(0x7f77580b1b00 legacy=0x7f77581dad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:01.829 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.828+0000 7f7760233640 1 -- 172.21.15.6:0/2730291643 shutdown_connections 2024-03-25T19:17:01.829 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.828+0000 7f7760233640 1 -- 172.21.15.6:0/2730291643 >> 172.21.15.6:0/2730291643 conn(0x7f77580617a0 msgr2=0x7f77580b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:17:01.829 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.828+0000 7f7760233640 1 -- 172.21.15.6:0/2730291643 shutdown_connections 2024-03-25T19:17:01.829 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.828+0000 7f7760233640 1 -- 172.21.15.6:0/2730291643 wait complete. 2024-03-25T19:17:01.829 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:01.828+0000 7f7760233640 1 librados: shutdown 2024-03-25T19:17:01.839 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:17:02.670 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:03:20.017 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:17:02.688 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:03:20.035 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:17:02.725 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:03:20.072 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:17:02.739 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:03:20.086 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:17:02.746 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:03:20.064 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:17:02.749 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:03:20.096 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:17:02.751 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:03:20.069 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:17:02.774 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:03:20.121 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:17:02.775 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:03:20.093 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:17:08.645 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:17:08.766 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.766+0000 7fb87f493640 10 monclient: get_monmap_and_config 2024-03-25T19:17:08.766 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.766+0000 7fb87f493640 10 monclient: build_initial_monmap 2024-03-25T19:17:08.766 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.766+0000 7fb87f493640 10 monclient: monmap: 2024-03-25T19:17:08.766 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:17:08.766 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:08.766 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:17:08.767175+0000 2024-03-25T19:17:08.766 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:17:08.767175+0000 2024-03-25T19:17:08.766 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:17:08.767 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:08.767 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:17:08.767 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:17:08.767 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:17:08.767 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:08.768 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.766+0000 7fb87f493640 1 Processor -- start 2024-03-25T19:17:08.768 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.767+0000 7fb87f493640 1 -- start start 2024-03-25T19:17:08.768 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.767+0000 7fb87f493640 10 monclient: init 2024-03-25T19:17:08.768 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.767+0000 7fb87f493640 10 monclient: _reopen_session rank -1 2024-03-25T19:17:08.768 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.767+0000 7fb87f493640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:17:08.769 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.767+0000 7fb87f493640 10 monclient(hunting): picked mon.noname-b con 0x7fb8780b4ff0 addr v1:172.21.15.6:6790/0 2024-03-25T19:17:08.769 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.767+0000 7fb87f493640 10 monclient(hunting): picked mon.noname-a con 0x7fb8780b4670 addr v1:172.21.15.6:6789/0 2024-03-25T19:17:08.769 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.767+0000 7fb87f493640 10 monclient(hunting): picked mon.noname-c con 0x7fb87813c480 addr v1:172.21.15.192:6789/0 2024-03-25T19:17:08.769 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.767+0000 7fb87f493640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8780ab0e0 con 0x7fb8780b4670 2024-03-25T19:17:08.769 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.767+0000 7fb87f493640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8780ab3c0 con 0x7fb87813c480 2024-03-25T19:17:08.769 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.767+0000 7fb87f493640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8780a9f30 con 0x7fb8780b4ff0 2024-03-25T19:17:08.769 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.767+0000 7fb87f493640 10 monclient(hunting): _renew_subs 2024-03-25T19:17:08.769 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.767+0000 7fb87f493640 10 monclient(hunting): authenticate will time out at 6133.007573s 2024-03-25T19:17:08.779 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.779+0000 7fb87ca2e640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb8780b4670 0x7fb8781425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52714/0 (socket says 172.21.15.6:52714) 2024-03-25T19:17:08.779 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.779+0000 7fb87ca2e640 1 -- 172.21.15.6:0/1742276210 learned_addr learned my addr 172.21.15.6:0/1742276210 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:17:08.790 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.789+0000 7fb86ffff640 1 -- 172.21.15.6:0/1742276210 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2858477466 0 0) 0x7fb8780ab3c0 con 0x7fb87813c480 2024-03-25T19:17:08.790 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.790+0000 7fb86ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:17:08.790 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.790+0000 7fb86ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:08.790 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.790+0000 7fb86ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:08.790 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.790+0000 7fb86ffff640 10 monclient(hunting): my global_id is 4357 2024-03-25T19:17:08.790 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.790+0000 7fb86ffff640 1 -- 172.21.15.6:0/1742276210 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb864003520 con 0x7fb87813c480 2024-03-25T19:17:08.793 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.793+0000 7fb86ffff640 1 -- 172.21.15.6:0/1742276210 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 62353546 0 0) 0x7fb8780a9f30 con 0x7fb8780b4ff0 2024-03-25T19:17:08.793 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.793+0000 7fb86ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:17:08.793 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.793+0000 7fb86ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:08.793 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.793+0000 7fb86ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:08.793 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.793+0000 7fb86ffff640 10 monclient(hunting): my global_id is 4355 2024-03-25T19:17:08.793 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.793+0000 7fb86ffff640 1 -- 172.21.15.6:0/1742276210 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8780ab3c0 con 0x7fb8780b4ff0 2024-03-25T19:17:08.799 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.799+0000 7fb86ffff640 1 -- 172.21.15.6:0/1742276210 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2802016537 0 0) 0x7fb8780ab0e0 con 0x7fb8780b4670 2024-03-25T19:17:08.799 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.799+0000 7fb86ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:17:08.799 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.799+0000 7fb86ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:08.799 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.799+0000 7fb86ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:08.799 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.799+0000 7fb86ffff640 10 monclient(hunting): my global_id is 4353 2024-03-25T19:17:08.799 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.799+0000 7fb86ffff640 1 -- 172.21.15.6:0/1742276210 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8780a9f30 con 0x7fb8780b4670 2024-03-25T19:17:08.808 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.807+0000 7fb86ffff640 1 -- 172.21.15.6:0/1742276210 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1205707061 0 0) 0x7fb864003520 con 0x7fb87813c480 2024-03-25T19:17:08.808 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.807+0000 7fb86ffff640 1 -- 172.21.15.6:0/1742276210 >> v1:172.21.15.6:6790/0 conn(0x7fb8780b4ff0 legacy=0x7fb87813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:08.808 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.807+0000 7fb86ffff640 1 -- 172.21.15.6:0/1742276210 >> v1:172.21.15.6:6789/0 conn(0x7fb8780b4670 legacy=0x7fb8781425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:08.808 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.807+0000 7fb86ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:17:08.808 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.807+0000 7fb86ffff640 1 monclient: found mon.noname-c 2024-03-25T19:17:08.808 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.807+0000 7fb86ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:17:08.808 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.807+0000 7fb86ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:17:08.808 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.807+0000 7fb86ffff640 1 -- 172.21.15.6:0/1742276210 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8780a9b70 con 0x7fb87813c480 2024-03-25T19:17:08.808 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.807+0000 7fb86ffff640 10 monclient: _finish_auth 0 2024-03-25T19:17:08.808 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.807+0000 7fb86ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:17:08.809 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.807+0000 7fb86ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:17:08.809 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.807+0000 7fb86ffff640 1 -- 172.21.15.6:0/1742276210 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fb870001a10 con 0x7fb87813c480 2024-03-25T19:17:08.809 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.807+0000 7fb86ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:17:08.809 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.807+0000 7fb86ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:17:08.809 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.807+0000 7fb86ffff640 10 monclient: dump: 2024-03-25T19:17:08.809 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:17:08.809 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:08.809 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:08.809 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:08.809 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:17:08.809 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:08.809 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:17:08.809 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:17:08.809 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:17:08.810 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:08.810 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.807+0000 7fb87f493640 5 monclient: authenticate success, global_id 4357 2024-03-25T19:17:08.810 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.807+0000 7fb87f493640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:17:08.819 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.819+0000 7fb86ffff640 1 -- 172.21.15.6:0/1742276210 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb870001b70 con 0x7fb87813c480 2024-03-25T19:17:08.819 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.819+0000 7fb86ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:17:08.819 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.819+0000 7fb86ffff640 1 -- 172.21.15.6:0/1742276210 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fb870001f80 con 0x7fb87813c480 2024-03-25T19:17:08.819 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.819+0000 7fb87f493640 10 monclient: get_monmap_and_config success 2024-03-25T19:17:08.819 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.819+0000 7fb87f493640 4 set_mon_vals no callback set 2024-03-25T19:17:08.819 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.819+0000 7fb86ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:17:08.819 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.819+0000 7fb86ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:17:08.820 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.819+0000 7fb86ffff640 10 monclient: dump: 2024-03-25T19:17:08.820 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:17:08.820 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:08.820 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:08.820 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:08.820 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:17:08.820 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:08.820 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:17:08.820 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:17:08.820 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:17:08.820 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:08.822 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.820+0000 7fb87f493640 10 monclient: shutdown 2024-03-25T19:17:08.822 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.820+0000 7fb87f493640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:17:08.822 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.820+0000 7fb87f493640 1 -- 172.21.15.6:0/1742276210 >> v1:172.21.15.192:6789/0 conn(0x7fb87813c480 legacy=0x7fb87814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:08.822 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.820+0000 7fb87f493640 1 -- 172.21.15.6:0/1742276210 shutdown_connections 2024-03-25T19:17:08.823 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.820+0000 7fb87f493640 1 -- 172.21.15.6:0/1742276210 >> 172.21.15.6:0/1742276210 conn(0x7fb8780617a0 msgr2=0x7fb878061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:17:08.823 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.820+0000 7fb87f493640 1 -- 172.21.15.6:0/1742276210 shutdown_connections 2024-03-25T19:17:08.823 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.820+0000 7fb87f493640 1 -- 172.21.15.6:0/1742276210 wait complete. 2024-03-25T19:17:08.823 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.821+0000 7fb87f493640 10 monclient: build_initial_monmap 2024-03-25T19:17:08.823 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.821+0000 7fb87f493640 10 monclient: monmap: 2024-03-25T19:17:08.823 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:17:08.823 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:17:08.823 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:17:08.823 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:17:08.823 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:17:08.823 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:17:08.823 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:17:08.824 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:17:08.824 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:17:08.824 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:08.824 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.821+0000 7fb87f493640 1 librados: starting msgr at 2024-03-25T19:17:08.824 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.821+0000 7fb87f493640 1 librados: starting objecter 2024-03-25T19:17:08.824 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.821+0000 7fb87f493640 1 Processor -- start 2024-03-25T19:17:08.824 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.821+0000 7fb87f493640 1 -- start start 2024-03-25T19:17:08.824 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.821+0000 7fb87f493640 1 librados: setting wanted keys 2024-03-25T19:17:08.824 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.821+0000 7fb87f493640 1 librados: calling monclient init 2024-03-25T19:17:08.824 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.821+0000 7fb87f493640 10 monclient: init 2024-03-25T19:17:08.824 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.822+0000 7fb87f493640 10 monclient: _reopen_session rank -1 2024-03-25T19:17:08.824 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.822+0000 7fb87f493640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:17:08.824 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.822+0000 7fb87f493640 10 monclient(hunting): picked mon.noname-b con 0x7fb8780b4670 addr v1:172.21.15.192:6789/0 2024-03-25T19:17:08.824 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.822+0000 7fb87f493640 10 monclient(hunting): picked mon.noname-a con 0x7fb8780b4ff0 addr v1:172.21.15.6:6789/0 2024-03-25T19:17:08.824 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.822+0000 7fb87f493640 10 monclient(hunting): picked mon.noname-c con 0x7fb87813c480 addr v1:172.21.15.6:6790/0 2024-03-25T19:17:08.824 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.822+0000 7fb87f493640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb87813dd70 con 0x7fb8780b4ff0 2024-03-25T19:17:08.825 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.822+0000 7fb87f493640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb878144440 con 0x7fb8780b4670 2024-03-25T19:17:08.825 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.822+0000 7fb87f493640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb87813dfe0 con 0x7fb87813c480 2024-03-25T19:17:08.825 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.822+0000 7fb87f493640 10 monclient(hunting): _renew_subs 2024-03-25T19:17:08.825 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.822+0000 7fb87f493640 10 monclient(hunting): authenticate will time out at 6133.062204s 2024-03-25T19:17:08.826 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.826+0000 7fb87d22f640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fb8780b4670 0x7fb878143620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:45932/0 (socket says 172.21.15.6:45932) 2024-03-25T19:17:08.826 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.826+0000 7fb87d22f640 1 -- 172.21.15.6:0/3694861845 learned_addr learned my addr 172.21.15.6:0/3694861845 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:17:08.830 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.831+0000 7fb86dffb640 10 client.?.objecter ms_handle_connect 0x7fb8780b4670 2024-03-25T19:17:08.831 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.831+0000 7fb86dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:17:08.838 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.838+0000 7fb86dffb640 1 -- 172.21.15.6:0/3694861845 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 137773474 0 0) 0x7fb878144440 con 0x7fb8780b4670 2024-03-25T19:17:08.838 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.838+0000 7fb86dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:17:08.838 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.838+0000 7fb86dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:08.838 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.838+0000 7fb86dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:08.838 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.838+0000 7fb86dffb640 10 monclient(hunting): my global_id is 4360 2024-03-25T19:17:08.838 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.838+0000 7fb86dffb640 1 -- 172.21.15.6:0/3694861845 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb85c004eb0 con 0x7fb8780b4670 2024-03-25T19:17:08.844 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.844+0000 7fb86dffb640 10 client.?.objecter ms_handle_connect 0x7fb8780b4ff0 2024-03-25T19:17:08.844 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.844+0000 7fb86dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:17:08.847 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.847+0000 7fb86dffb640 10 client.?.objecter ms_handle_connect 0x7fb87813c480 2024-03-25T19:17:08.847 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.847+0000 7fb86dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:17:08.850 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.850+0000 7fb86dffb640 1 -- 172.21.15.6:0/3694861845 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3092579314 0 0) 0x7fb85c004eb0 con 0x7fb8780b4670 2024-03-25T19:17:08.850 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.850+0000 7fb86dffb640 1 -- 172.21.15.6:0/3694861845 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb878144440 con 0x7fb8780b4670 2024-03-25T19:17:08.851 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.850+0000 7fb86dffb640 1 -- 172.21.15.6:0/3694861845 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fb874001a30 con 0x7fb8780b4670 2024-03-25T19:17:08.851 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.850+0000 7fb86dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:17:08.851 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.850+0000 7fb86dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:17:08.851 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.850+0000 7fb86dffb640 10 monclient(hunting): dump: 2024-03-25T19:17:08.851 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:17:08.851 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:08.851 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:08.851 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:08.851 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:17:08.851 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:08.851 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:17:08.851 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:17:08.851 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:17:08.851 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:08.851 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.850+0000 7fb86dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:17:08.858 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.858+0000 7fb86dffb640 1 -- 172.21.15.6:0/3694861845 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1072961754 0 0) 0x7fb87813dd70 con 0x7fb8780b4ff0 2024-03-25T19:17:08.858 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.858+0000 7fb86dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:17:08.858 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.858+0000 7fb86dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:08.858 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.858+0000 7fb86dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:08.858 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.858+0000 7fb86dffb640 10 monclient(hunting): my global_id is 4356 2024-03-25T19:17:08.858 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.858+0000 7fb86dffb640 1 -- 172.21.15.6:0/3694861845 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb85c004eb0 con 0x7fb8780b4ff0 2024-03-25T19:17:08.864 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.863+0000 7fb86dffb640 1 -- 172.21.15.6:0/3694861845 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2021151123 0 0) 0x7fb878144440 con 0x7fb8780b4670 2024-03-25T19:17:08.864 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.863+0000 7fb86dffb640 1 -- 172.21.15.6:0/3694861845 >> v1:172.21.15.6:6790/0 conn(0x7fb87813c480 legacy=0x7fb8781e2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:08.864 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.863+0000 7fb86dffb640 1 -- 172.21.15.6:0/3694861845 >> v1:172.21.15.6:6789/0 conn(0x7fb8780b4ff0 legacy=0x7fb878143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:08.864 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.863+0000 7fb86dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:17:08.864 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.863+0000 7fb86dffb640 1 monclient: found mon.b 2024-03-25T19:17:08.864 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.863+0000 7fb86dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:17:08.864 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.863+0000 7fb86dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:17:08.864 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.863+0000 7fb86dffb640 1 -- 172.21.15.6:0/3694861845 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87813c910 con 0x7fb8780b4670 2024-03-25T19:17:08.864 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.863+0000 7fb86dffb640 10 monclient: _finish_auth 0 2024-03-25T19:17:08.864 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.863+0000 7fb86dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:17:08.864 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.863+0000 7fb86dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:17:08.865 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.863+0000 7fb87f493640 5 monclient: authenticate success, global_id 4360 2024-03-25T19:17:08.865 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.863+0000 7fb87f493640 10 monclient: _renew_subs 2024-03-25T19:17:08.865 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.863+0000 7fb87f493640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:17:08.865 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.863+0000 7fb87f493640 1 -- 172.21.15.6:0/3694861845 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb878144a60 con 0x7fb8780b4670 2024-03-25T19:17:08.865 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.863+0000 7fb87f493640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:17:08.865 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.863+0000 7fb87f493640 10 monclient: _renew_subs 2024-03-25T19:17:08.865 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.863+0000 7fb87f493640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:17:08.865 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.863+0000 7fb87f493640 1 -- 172.21.15.6:0/3694861845 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb878145d20 con 0x7fb8780b4670 2024-03-25T19:17:08.865 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.863+0000 7fb87f493640 1 librados: init done 2024-03-25T19:17:08.866 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.864+0000 7fb87f493640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:17:08.866 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.864+0000 7fb87f493640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:17:08.866 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.864+0000 7fb87f493640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:17:08.866 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:08.864+0000 7fb87f493640 1 -- 172.21.15.6:0/3694861845 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb87806dbd0 con 0x7fb8780b4670 2024-03-25T19:17:09.086 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:09.086+0000 7fb86dffb640 1 -- 172.21.15.6:0/3694861845 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb874001cd0 con 0x7fb8780b4670 2024-03-25T19:17:09.086 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:09.086+0000 7fb86dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:17:09.086 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:09.086+0000 7fb86f7fe640 4 set_mon_vals no callback set 2024-03-25T19:17:09.156 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:09.156+0000 7fb86dffb640 1 -- 172.21.15.6:0/3694861845 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fb874003a60 con 0x7fb8780b4670 2024-03-25T19:17:09.156 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:09.156+0000 7fb86dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:17:09.156 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:09.156+0000 7fb86dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:17:09.157 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:09.156+0000 7fb86dffb640 10 monclient: dump: 2024-03-25T19:17:09.157 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:17:09.157 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:09.157 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:09.157 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:09.157 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:17:09.157 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:09.157 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:17:09.157 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:17:09.157 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:17:09.157 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:09.158 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:09.157+0000 7fb86dffb640 1 -- 172.21.15.6:0/3694861845 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7fb874003c40 con 0x7fb8780b4670 2024-03-25T19:17:09.158 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:09.158+0000 7fb87ca2e640 10 monclient: get_auth_request con 0x7fb85c06de20 auth_method 0 2024-03-25T19:17:09.158 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:09.158+0000 7fb86dffb640 10 client.4360.objecter ms_handle_connect 0x7fb85c06de20 2024-03-25T19:17:09.162 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:09.162+0000 7fb86dffb640 1 -- 172.21.15.6:0/3694861845 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7fb874054900 con 0x7fb8780b4670 2024-03-25T19:17:09.162 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:09.162+0000 7fb86dffb640 10 client.4360.objecter ms_dispatch 0x7fb878000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:17:09.162 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:09.162+0000 7fb86dffb640 3 client.4360.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:17:09.162 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:09.162+0000 7fb86dffb640 3 client.4360.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:17:09.162 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:09.162+0000 7fb86dffb640 20 client.4360.objecter dump_active .. 0 homeless 2024-03-25T19:17:09.169 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:09.169+0000 7fb86dffb640 1 -- 172.21.15.6:0/3694861845 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb874050410 con 0x7fb8780b4670 2024-03-25T19:17:09.169 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:09.169+0000 7fb86dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:17:09.169 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:09.169+0000 7fb86dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:17:09.401 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:09.401+0000 7fb87f493640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:17:09.401 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:09.401+0000 7fb87f493640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:17:09.401 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:09.401+0000 7fb87f493640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:17:09.401 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:09.401+0000 7fb87f493640 1 -- 172.21.15.6:0/3694861845 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb87813caa0 con 0x7fb8780b4670 2024-03-25T19:17:09.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:09.471+0000 7fb86dffb640 1 -- 172.21.15.6:0/3694861845 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7fb874052340 con 0x7fb8780b4670 2024-03-25T19:17:09.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:09.471+0000 7fb86dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:17:09.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:09.471+0000 7fb86dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:17:09.471 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:17:09.471 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:17:09.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:09.473+0000 7fb87f493640 10 librados: watch_flush enter 2024-03-25T19:17:09.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:09.473+0000 7fb87f493640 10 librados: watch_flush exit 2024-03-25T19:17:09.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:09.473+0000 7fb87f493640 20 client.4360.objecter shutdown clearing up homeless session... 2024-03-25T19:17:09.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:09.473+0000 7fb87f493640 10 client.4360.objecter successfully canceled tick 2024-03-25T19:17:09.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:09.474+0000 7fb87f493640 1 -- 172.21.15.6:0/3694861845 >> v1:172.21.15.6:6800/2936652650 conn(0x7fb85c06de20 legacy=0x7fb85c0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:09.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:09.474+0000 7fb87f493640 10 monclient: shutdown 2024-03-25T19:17:09.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:09.474+0000 7fb87f493640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:17:09.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:09.474+0000 7fb87f493640 1 -- 172.21.15.6:0/3694861845 >> v1:172.21.15.192:6789/0 conn(0x7fb8780b4670 legacy=0x7fb878143620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:09.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:09.474+0000 7fb87f493640 1 -- 172.21.15.6:0/3694861845 shutdown_connections 2024-03-25T19:17:09.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:09.474+0000 7fb87f493640 1 -- 172.21.15.6:0/3694861845 >> 172.21.15.6:0/3694861845 conn(0x7fb8780617a0 msgr2=0x7fb8781407f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:17:09.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:09.474+0000 7fb87f493640 1 -- 172.21.15.6:0/3694861845 shutdown_connections 2024-03-25T19:17:09.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:09.474+0000 7fb87f493640 1 -- 172.21.15.6:0/3694861845 wait complete. 2024-03-25T19:17:09.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:09.474+0000 7fb87f493640 1 librados: shutdown 2024-03-25T19:17:09.485 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:17:16.294 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:17:16.414 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.413+0000 7f8a69f32640 10 monclient: get_monmap_and_config 2024-03-25T19:17:16.414 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.414+0000 7f8a69f32640 10 monclient: build_initial_monmap 2024-03-25T19:17:16.414 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.414+0000 7f8a69f32640 10 monclient: monmap: 2024-03-25T19:17:16.414 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:17:16.414 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:16.414 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:17:16.415003+0000 2024-03-25T19:17:16.414 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:17:16.415003+0000 2024-03-25T19:17:16.414 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:17:16.415 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:16.415 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:17:16.415 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:17:16.415 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:17:16.415 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:16.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.415+0000 7f8a69f32640 1 Processor -- start 2024-03-25T19:17:16.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.415+0000 7f8a69f32640 1 -- start start 2024-03-25T19:17:16.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.415+0000 7f8a69f32640 10 monclient: init 2024-03-25T19:17:16.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.416+0000 7f8a69f32640 10 monclient: _reopen_session rank -1 2024-03-25T19:17:16.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.416+0000 7f8a69f32640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:17:16.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.416+0000 7f8a69f32640 10 monclient(hunting): picked mon.noname-a con 0x7f8a640b4e70 addr v1:172.21.15.6:6789/0 2024-03-25T19:17:16.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.416+0000 7f8a69f32640 10 monclient(hunting): picked mon.noname-c con 0x7f8a640b44b0 addr v1:172.21.15.192:6789/0 2024-03-25T19:17:16.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.416+0000 7f8a69f32640 10 monclient(hunting): picked mon.noname-b con 0x7f8a640b1b00 addr v1:172.21.15.6:6790/0 2024-03-25T19:17:16.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.416+0000 7f8a69f32640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a640a96f0 con 0x7f8a640b4e70 2024-03-25T19:17:16.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.416+0000 7f8a69f32640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a640ab0e0 con 0x7f8a640b44b0 2024-03-25T19:17:16.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.416+0000 7f8a69f32640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a64145f70 con 0x7f8a640b1b00 2024-03-25T19:17:16.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.416+0000 7f8a69f32640 10 monclient(hunting): _renew_subs 2024-03-25T19:17:16.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.416+0000 7f8a69f32640 10 monclient(hunting): authenticate will time out at 6140.655945s 2024-03-25T19:17:16.421 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.421+0000 7f8a63fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f8a640b1b00 0x7f8a6414dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:53302/0 (socket says 172.21.15.6:53302) 2024-03-25T19:17:16.421 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.421+0000 7f8a63fff640 1 -- 172.21.15.6:0/1065543767 learned_addr learned my addr 172.21.15.6:0/1065543767 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:17:16.437 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.436+0000 7f8a627fc640 1 -- 172.21.15.6:0/1065543767 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2167226100 0 0) 0x7f8a64145f70 con 0x7f8a640b1b00 2024-03-25T19:17:16.437 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.436+0000 7f8a627fc640 10 monclient(hunting): _negotiate 2024-03-25T19:17:16.437 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.436+0000 7f8a627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:16.437 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.436+0000 7f8a627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:16.437 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.436+0000 7f8a627fc640 10 monclient(hunting): my global_id is 4361 2024-03-25T19:17:16.437 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.436+0000 7f8a627fc640 1 -- 172.21.15.6:0/1065543767 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a4c003520 con 0x7f8a640b1b00 2024-03-25T19:17:16.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.444+0000 7f8a627fc640 1 -- 172.21.15.6:0/1065543767 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3101741337 0 0) 0x7f8a640a96f0 con 0x7f8a640b4e70 2024-03-25T19:17:16.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.444+0000 7f8a627fc640 10 monclient(hunting): _negotiate 2024-03-25T19:17:16.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.444+0000 7f8a627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:16.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.444+0000 7f8a627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:16.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.444+0000 7f8a627fc640 10 monclient(hunting): my global_id is 4359 2024-03-25T19:17:16.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.444+0000 7f8a627fc640 1 -- 172.21.15.6:0/1065543767 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a64145f70 con 0x7f8a640b4e70 2024-03-25T19:17:16.457 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.457+0000 7f8a627fc640 1 -- 172.21.15.6:0/1065543767 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2571075393 0 0) 0x7f8a640ab0e0 con 0x7f8a640b44b0 2024-03-25T19:17:16.458 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.457+0000 7f8a627fc640 10 monclient(hunting): _negotiate 2024-03-25T19:17:16.458 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.457+0000 7f8a627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:16.458 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.457+0000 7f8a627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:16.458 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.457+0000 7f8a627fc640 10 monclient(hunting): my global_id is 4363 2024-03-25T19:17:16.458 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.457+0000 7f8a627fc640 1 -- 172.21.15.6:0/1065543767 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a640a96f0 con 0x7f8a640b44b0 2024-03-25T19:17:16.463 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.462+0000 7f8a627fc640 1 -- 172.21.15.6:0/1065543767 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4108700375 0 0) 0x7f8a64145f70 con 0x7f8a640b4e70 2024-03-25T19:17:16.463 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.462+0000 7f8a627fc640 1 -- 172.21.15.6:0/1065543767 >> v1:172.21.15.6:6790/0 conn(0x7f8a640b1b00 legacy=0x7f8a6414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:16.463 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.462+0000 7f8a627fc640 1 -- 172.21.15.6:0/1065543767 >> v1:172.21.15.192:6789/0 conn(0x7f8a640b44b0 legacy=0x7f8a64145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:16.463 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.462+0000 7f8a627fc640 10 monclient: _finish_hunting 0 2024-03-25T19:17:16.463 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.462+0000 7f8a627fc640 1 monclient: found mon.noname-a 2024-03-25T19:17:16.463 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.462+0000 7f8a627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:17:16.463 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.462+0000 7f8a627fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-03-25T19:17:16.463 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.462+0000 7f8a627fc640 1 -- 172.21.15.6:0/1065543767 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a640a9b80 con 0x7f8a640b4e70 2024-03-25T19:17:16.463 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.462+0000 7f8a627fc640 10 monclient: _finish_auth 0 2024-03-25T19:17:16.463 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.462+0000 7f8a627fc640 10 monclient: _check_auth_tickets 2024-03-25T19:17:16.463 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.462+0000 7f8a627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:17:16.464 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.462+0000 7f8a627fc640 1 -- 172.21.15.6:0/1065543767 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f8a54001670 con 0x7f8a640b4e70 2024-03-25T19:17:16.464 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.462+0000 7f8a69f32640 5 monclient: authenticate success, global_id 4359 2024-03-25T19:17:16.464 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.462+0000 7f8a69f32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:17:16.464 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.462+0000 7f8a627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:17:16.464 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.462+0000 7f8a627fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:17:16.464 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.462+0000 7f8a627fc640 10 monclient: dump: 2024-03-25T19:17:16.464 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:17:16.464 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:16.464 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:16.464 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:16.464 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:17:16.464 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:16.464 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:17:16.465 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:17:16.465 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:17:16.465 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:16.465 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.462+0000 7f8a69f32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:17:16.491 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.490+0000 7f8a627fc640 1 -- 172.21.15.6:0/1065543767 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a540017d0 con 0x7f8a640b4e70 2024-03-25T19:17:16.491 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.491+0000 7f8a627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:17:16.491 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.491+0000 7f8a627fc640 1 -- 172.21.15.6:0/1065543767 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f8a54001be0 con 0x7f8a640b4e70 2024-03-25T19:17:16.491 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.491+0000 7f8a627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:17:16.491 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.491+0000 7f8a627fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:17:16.491 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.491+0000 7f8a627fc640 10 monclient: dump: 2024-03-25T19:17:16.491 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:17:16.491 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:16.491 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:16.491 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:16.491 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:17:16.492 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:16.492 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:17:16.492 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:17:16.492 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:17:16.492 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:16.492 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.491+0000 7f8a69f32640 10 monclient: get_monmap_and_config success 2024-03-25T19:17:16.492 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.491+0000 7f8a69f32640 4 set_mon_vals no callback set 2024-03-25T19:17:16.494 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.491+0000 7f8a69f32640 10 monclient: shutdown 2024-03-25T19:17:16.494 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.491+0000 7f8a69f32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:17:16.495 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.491+0000 7f8a69f32640 1 -- 172.21.15.6:0/1065543767 >> v1:172.21.15.6:6789/0 conn(0x7f8a640b4e70 legacy=0x7f8a6413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:16.495 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.491+0000 7f8a69f32640 1 -- 172.21.15.6:0/1065543767 shutdown_connections 2024-03-25T19:17:16.495 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.491+0000 7f8a69f32640 1 -- 172.21.15.6:0/1065543767 >> 172.21.15.6:0/1065543767 conn(0x7f8a640617a0 msgr2=0x7f8a64061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:17:16.495 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.491+0000 7f8a69f32640 1 -- 172.21.15.6:0/1065543767 shutdown_connections 2024-03-25T19:17:16.495 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.491+0000 7f8a69f32640 1 -- 172.21.15.6:0/1065543767 wait complete. 2024-03-25T19:17:16.495 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.492+0000 7f8a69f32640 10 monclient: build_initial_monmap 2024-03-25T19:17:16.495 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.492+0000 7f8a69f32640 10 monclient: monmap: 2024-03-25T19:17:16.495 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:17:16.497 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:17:16.497 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:17:16.497 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:17:16.498 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:17:16.498 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:17:16.498 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:17:16.498 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:17:16.498 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:17:16.498 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:16.498 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.493+0000 7f8a69f32640 1 librados: starting msgr at 2024-03-25T19:17:16.498 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.493+0000 7f8a69f32640 1 librados: starting objecter 2024-03-25T19:17:16.498 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.493+0000 7f8a69f32640 1 Processor -- start 2024-03-25T19:17:16.498 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.493+0000 7f8a69f32640 1 -- start start 2024-03-25T19:17:16.498 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.493+0000 7f8a69f32640 1 librados: setting wanted keys 2024-03-25T19:17:16.499 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.493+0000 7f8a69f32640 1 librados: calling monclient init 2024-03-25T19:17:16.499 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.493+0000 7f8a69f32640 10 monclient: init 2024-03-25T19:17:16.499 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.494+0000 7f8a69f32640 10 monclient: _reopen_session rank -1 2024-03-25T19:17:16.499 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.494+0000 7f8a69f32640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:17:16.499 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.494+0000 7f8a69f32640 10 monclient(hunting): picked mon.noname-b con 0x7f8a640b1b00 addr v1:172.21.15.192:6789/0 2024-03-25T19:17:16.499 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.494+0000 7f8a69f32640 10 monclient(hunting): picked mon.noname-a con 0x7f8a640b44b0 addr v1:172.21.15.6:6789/0 2024-03-25T19:17:16.499 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.494+0000 7f8a69f32640 10 monclient(hunting): picked mon.noname-c con 0x7f8a640b4e70 addr v1:172.21.15.6:6790/0 2024-03-25T19:17:16.499 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.494+0000 7f8a69f32640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a6413ba00 con 0x7f8a640b44b0 2024-03-25T19:17:16.499 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.494+0000 7f8a69f32640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a6413b570 con 0x7f8a640b1b00 2024-03-25T19:17:16.499 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.494+0000 7f8a69f32640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a6413b700 con 0x7f8a640b4e70 2024-03-25T19:17:16.499 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.494+0000 7f8a69f32640 10 monclient(hunting): _renew_subs 2024-03-25T19:17:16.499 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.494+0000 7f8a69f32640 10 monclient(hunting): authenticate will time out at 6140.734150s 2024-03-25T19:17:16.505 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.505+0000 7f8a637fe640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f8a640b1b00 0x7f8a6413ae60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:50578/0 (socket says 172.21.15.6:50578) 2024-03-25T19:17:16.505 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.505+0000 7f8a637fe640 1 -- 172.21.15.6:0/3862184722 learned_addr learned my addr 172.21.15.6:0/3862184722 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:17:16.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.506+0000 7f8a4bfff640 10 client.?.objecter ms_handle_connect 0x7f8a640b1b00 2024-03-25T19:17:16.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.506+0000 7f8a4bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:17:16.510 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.511+0000 7f8a4bfff640 10 client.?.objecter ms_handle_connect 0x7f8a640b44b0 2024-03-25T19:17:16.511 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.511+0000 7f8a4bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:17:16.513 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.513+0000 7f8a4bfff640 10 client.?.objecter ms_handle_connect 0x7f8a640b4e70 2024-03-25T19:17:16.513 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.513+0000 7f8a4bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:17:16.517 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.517+0000 7f8a4bfff640 1 -- 172.21.15.6:0/3862184722 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3747169751 0 0) 0x7f8a6413b570 con 0x7f8a640b1b00 2024-03-25T19:17:16.518 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.517+0000 7f8a4bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:17:16.518 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.517+0000 7f8a4bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:16.518 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.517+0000 7f8a4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:16.518 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.517+0000 7f8a4bfff640 10 monclient(hunting): my global_id is 4366 2024-03-25T19:17:16.518 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.517+0000 7f8a4bfff640 1 -- 172.21.15.6:0/3862184722 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a50004eb0 con 0x7f8a640b1b00 2024-03-25T19:17:16.521 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.521+0000 7f8a4bfff640 1 -- 172.21.15.6:0/3862184722 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1562519488 0 0) 0x7f8a6413ba00 con 0x7f8a640b44b0 2024-03-25T19:17:16.521 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.521+0000 7f8a4bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:17:16.522 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.521+0000 7f8a4bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:16.522 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.521+0000 7f8a4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:16.522 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.521+0000 7f8a4bfff640 10 monclient(hunting): my global_id is 4362 2024-03-25T19:17:16.522 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.521+0000 7f8a4bfff640 1 -- 172.21.15.6:0/3862184722 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a6413b570 con 0x7f8a640b44b0 2024-03-25T19:17:16.531 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.530+0000 7f8a4bfff640 1 -- 172.21.15.6:0/3862184722 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 146565434 0 0) 0x7f8a6413b700 con 0x7f8a640b4e70 2024-03-25T19:17:16.531 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.530+0000 7f8a4bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:17:16.531 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.530+0000 7f8a4bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:16.531 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.530+0000 7f8a4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:16.531 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.530+0000 7f8a4bfff640 10 monclient(hunting): my global_id is 4364 2024-03-25T19:17:16.531 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.530+0000 7f8a4bfff640 1 -- 172.21.15.6:0/3862184722 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a6413ba00 con 0x7f8a640b4e70 2024-03-25T19:17:16.531 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.531+0000 7f8a4bfff640 1 -- 172.21.15.6:0/3862184722 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3618438992 0 0) 0x7f8a50004eb0 con 0x7f8a640b1b00 2024-03-25T19:17:16.531 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.531+0000 7f8a4bfff640 1 -- 172.21.15.6:0/3862184722 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a6413b700 con 0x7f8a640b1b00 2024-03-25T19:17:16.532 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.531+0000 7f8a4bfff640 1 -- 172.21.15.6:0/3862184722 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f8a54001670 con 0x7f8a640b1b00 2024-03-25T19:17:16.532 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.531+0000 7f8a4bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:17:16.532 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.531+0000 7f8a4bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:17:16.532 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.531+0000 7f8a4bfff640 10 monclient(hunting): dump: 2024-03-25T19:17:16.532 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:17:16.532 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:16.532 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:16.532 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:16.532 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:17:16.532 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:16.532 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:17:16.532 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:17:16.532 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:17:16.533 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:16.533 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.531+0000 7f8a4bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:17:16.537 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.537+0000 7f8a4bfff640 1 -- 172.21.15.6:0/3862184722 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2167392298 0 0) 0x7f8a6413b570 con 0x7f8a640b44b0 2024-03-25T19:17:16.537 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.537+0000 7f8a4bfff640 1 -- 172.21.15.6:0/3862184722 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a50004eb0 con 0x7f8a640b44b0 2024-03-25T19:17:16.537 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.537+0000 7f8a4bfff640 1 -- 172.21.15.6:0/3862184722 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f8a58001870 con 0x7f8a640b44b0 2024-03-25T19:17:16.538 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.537+0000 7f8a4bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:17:16.538 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.537+0000 7f8a4bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:17:16.538 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.537+0000 7f8a4bfff640 10 monclient(hunting): dump: 2024-03-25T19:17:16.538 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:17:16.538 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:16.538 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:16.538 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:16.538 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:17:16.538 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:16.538 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:17:16.538 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:17:16.538 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:17:16.538 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:16.550 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.550+0000 7f8a4bfff640 1 -- 172.21.15.6:0/3862184722 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1257633044 0 0) 0x7f8a6413b700 con 0x7f8a640b1b00 2024-03-25T19:17:16.550 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.550+0000 7f8a4bfff640 1 -- 172.21.15.6:0/3862184722 >> v1:172.21.15.6:6790/0 conn(0x7f8a640b4e70 legacy=0x7f8a64144540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:16.550 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.550+0000 7f8a4bfff640 1 -- 172.21.15.6:0/3862184722 >> v1:172.21.15.6:6789/0 conn(0x7f8a640b44b0 legacy=0x7f8a64143e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:16.550 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.550+0000 7f8a4bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:17:16.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.550+0000 7f8a4bfff640 1 monclient: found mon.b 2024-03-25T19:17:16.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.550+0000 7f8a4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:17:16.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.550+0000 7f8a4bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:17:16.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.550+0000 7f8a4bfff640 1 -- 172.21.15.6:0/3862184722 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a641454d0 con 0x7f8a640b1b00 2024-03-25T19:17:16.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.550+0000 7f8a4bfff640 10 monclient: _finish_auth 0 2024-03-25T19:17:16.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.550+0000 7f8a4bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:17:16.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.550+0000 7f8a4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:17:16.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.550+0000 7f8a69f32640 5 monclient: authenticate success, global_id 4366 2024-03-25T19:17:16.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.550+0000 7f8a69f32640 10 monclient: _renew_subs 2024-03-25T19:17:16.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.550+0000 7f8a69f32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:17:16.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.550+0000 7f8a69f32640 1 -- 172.21.15.6:0/3862184722 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a64145780 con 0x7f8a640b1b00 2024-03-25T19:17:16.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.550+0000 7f8a69f32640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:17:16.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.550+0000 7f8a69f32640 10 monclient: _renew_subs 2024-03-25T19:17:16.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.550+0000 7f8a69f32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:17:16.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.550+0000 7f8a69f32640 1 -- 172.21.15.6:0/3862184722 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a641e33b0 con 0x7f8a640b1b00 2024-03-25T19:17:16.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.550+0000 7f8a69f32640 1 librados: init done 2024-03-25T19:17:16.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.551+0000 7f8a69f32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:17:16.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.551+0000 7f8a69f32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:17:16.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.551+0000 7f8a69f32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:17:16.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.551+0000 7f8a69f32640 1 -- 172.21.15.6:0/3862184722 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a6404f730 con 0x7f8a640b1b00 2024-03-25T19:17:16.558 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.558+0000 7f8a4bfff640 1 -- 172.21.15.6:0/3862184722 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a540038a0 con 0x7f8a640b1b00 2024-03-25T19:17:16.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.558+0000 7f8a4bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:17:16.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.558+0000 7f8a61ffb640 4 set_mon_vals no callback set 2024-03-25T19:17:16.830 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.830+0000 7f8a4bfff640 1 -- 172.21.15.6:0/3862184722 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f8a54003cb0 con 0x7f8a640b1b00 2024-03-25T19:17:16.831 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.830+0000 7f8a4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:17:16.831 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.830+0000 7f8a4bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:17:16.831 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.830+0000 7f8a4bfff640 10 monclient: dump: 2024-03-25T19:17:16.831 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:17:16.831 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:16.831 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:16.831 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:16.831 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:17:16.831 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:16.831 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:17:16.832 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:17:16.832 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:17:16.832 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:16.832 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.831+0000 7f8a4bfff640 1 -- 172.21.15.6:0/3862184722 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f8a54003e90 con 0x7f8a640b1b00 2024-03-25T19:17:16.832 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.832+0000 7f8a62ffd640 10 monclient: get_auth_request con 0x7f8a5006df70 auth_method 0 2024-03-25T19:17:16.833 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.832+0000 7f8a4bfff640 10 client.4366.objecter ms_handle_connect 0x7f8a5006df70 2024-03-25T19:17:16.834 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.834+0000 7f8a4bfff640 1 -- 172.21.15.6:0/3862184722 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f8a54054940 con 0x7f8a640b1b00 2024-03-25T19:17:16.834 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.834+0000 7f8a4bfff640 10 client.4366.objecter ms_dispatch 0x7f8a64000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:17:16.834 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.834+0000 7f8a4bfff640 3 client.4366.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:17:16.834 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.834+0000 7f8a4bfff640 3 client.4366.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:17:16.834 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.834+0000 7f8a4bfff640 20 client.4366.objecter dump_active .. 0 homeless 2024-03-25T19:17:16.838 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.838+0000 7f8a4bfff640 1 -- 172.21.15.6:0/3862184722 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8a54050450 con 0x7f8a640b1b00 2024-03-25T19:17:16.838 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.838+0000 7f8a4bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:17:16.838 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:16.838+0000 7f8a4bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:17:17.074 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:17.074+0000 7f8a69f32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:17:17.074 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:17.074+0000 7f8a69f32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:17:17.074 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:17.074+0000 7f8a69f32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:17:17.074 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:17.074+0000 7f8a69f32640 1 -- 172.21.15.6:0/3862184722 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a640aa090 con 0x7f8a640b1b00 2024-03-25T19:17:17.100 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:17.100+0000 7f8a4bfff640 1 -- 172.21.15.6:0/3862184722 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f8a54052380 con 0x7f8a640b1b00 2024-03-25T19:17:17.100 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:17.100+0000 7f8a4bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:17:17.100 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:17.100+0000 7f8a4bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:17:17.101 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:17:17.101 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:17:17.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:17.103+0000 7f8a69f32640 10 librados: watch_flush enter 2024-03-25T19:17:17.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:17.103+0000 7f8a69f32640 10 librados: watch_flush exit 2024-03-25T19:17:17.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:17.103+0000 7f8a69f32640 20 client.4366.objecter shutdown clearing up homeless session... 2024-03-25T19:17:17.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:17.103+0000 7f8a69f32640 10 client.4366.objecter successfully canceled tick 2024-03-25T19:17:17.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:17.103+0000 7f8a69f32640 1 -- 172.21.15.6:0/3862184722 >> v1:172.21.15.6:6800/2936652650 conn(0x7f8a5006df70 legacy=0x7f8a50070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:17.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:17.103+0000 7f8a69f32640 10 monclient: shutdown 2024-03-25T19:17:17.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:17.103+0000 7f8a69f32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:17:17.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:17.103+0000 7f8a69f32640 1 -- 172.21.15.6:0/3862184722 >> v1:172.21.15.192:6789/0 conn(0x7f8a640b1b00 legacy=0x7f8a6413ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:17.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:17.103+0000 7f8a69f32640 1 -- 172.21.15.6:0/3862184722 shutdown_connections 2024-03-25T19:17:17.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:17.103+0000 7f8a69f32640 1 -- 172.21.15.6:0/3862184722 >> 172.21.15.6:0/3862184722 conn(0x7f8a640617a0 msgr2=0x7f8a640b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:17:17.105 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:17.104+0000 7f8a69f32640 1 -- 172.21.15.6:0/3862184722 shutdown_connections 2024-03-25T19:17:17.105 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:17.104+0000 7f8a69f32640 1 -- 172.21.15.6:0/3862184722 wait complete. 2024-03-25T19:17:17.105 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:17.104+0000 7f8a69f32640 1 librados: shutdown 2024-03-25T19:17:17.116 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:17:22.725 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:03:40.072 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:17:22.731 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:03:40.078 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:17:22.773 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:03:40.120 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:17:22.789 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:03:40.107 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:17:22.796 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:03:40.114 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:17:22.798 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:03:40.145 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:17:22.810 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:03:40.156 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:17:22.826 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:03:40.144 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:17:22.844 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:03:40.191 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:17:23.922 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:17:24.037 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.037+0000 7f880ee86640 10 monclient: get_monmap_and_config 2024-03-25T19:17:24.037 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.037+0000 7f880ee86640 10 monclient: build_initial_monmap 2024-03-25T19:17:24.037 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.037+0000 7f880ee86640 10 monclient: monmap: 2024-03-25T19:17:24.037 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:17:24.037 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:24.038 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:17:24.038571+0000 2024-03-25T19:17:24.038 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:17:24.038571+0000 2024-03-25T19:17:24.038 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:17:24.038 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:24.038 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:17:24.038 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:17:24.038 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:17:24.038 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:24.040 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.038+0000 7f880ee86640 1 Processor -- start 2024-03-25T19:17:24.040 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.038+0000 7f880ee86640 1 -- start start 2024-03-25T19:17:24.040 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.038+0000 7f880ee86640 10 monclient: init 2024-03-25T19:17:24.040 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.039+0000 7f880ee86640 10 monclient: _reopen_session rank -1 2024-03-25T19:17:24.040 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.039+0000 7f880ee86640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:17:24.040 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.039+0000 7f880ee86640 10 monclient(hunting): picked mon.noname-b con 0x7f88080b4e70 addr v1:172.21.15.6:6790/0 2024-03-25T19:17:24.040 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.039+0000 7f880ee86640 10 monclient(hunting): picked mon.noname-a con 0x7f88080b44b0 addr v1:172.21.15.6:6789/0 2024-03-25T19:17:24.040 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.039+0000 7f880ee86640 10 monclient(hunting): picked mon.noname-c con 0x7f88080b1b00 addr v1:172.21.15.192:6789/0 2024-03-25T19:17:24.040 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.039+0000 7f880ee86640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88080a96f0 con 0x7f88080b44b0 2024-03-25T19:17:24.040 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.039+0000 7f880ee86640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88080ab0e0 con 0x7f88080b1b00 2024-03-25T19:17:24.040 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.039+0000 7f880ee86640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8808145f70 con 0x7f88080b4e70 2024-03-25T19:17:24.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.039+0000 7f880ee86640 10 monclient(hunting): _renew_subs 2024-03-25T19:17:24.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.039+0000 7f880ee86640 10 monclient(hunting): authenticate will time out at 6148.279108s 2024-03-25T19:17:24.047 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.047+0000 7f880cc22640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f88080b4e70 0x7f880813f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:35226/0 (socket says 172.21.15.6:35226) 2024-03-25T19:17:24.047 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.047+0000 7f880cc22640 1 -- 172.21.15.6:0/541897271 learned_addr learned my addr 172.21.15.6:0/541897271 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:17:24.063 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.063+0000 7f88077fe640 1 -- 172.21.15.6:0/541897271 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3855024776 0 0) 0x7f8808145f70 con 0x7f88080b4e70 2024-03-25T19:17:24.063 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.063+0000 7f88077fe640 10 monclient(hunting): _negotiate 2024-03-25T19:17:24.063 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.063+0000 7f88077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:24.063 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.063+0000 7f88077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:24.064 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.063+0000 7f88077fe640 10 monclient(hunting): my global_id is 4367 2024-03-25T19:17:24.064 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.063+0000 7f88077fe640 1 -- 172.21.15.6:0/541897271 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87f8003520 con 0x7f88080b4e70 2024-03-25T19:17:24.069 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.069+0000 7f88077fe640 1 -- 172.21.15.6:0/541897271 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1198686210 0 0) 0x7f88080ab0e0 con 0x7f88080b1b00 2024-03-25T19:17:24.069 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.069+0000 7f88077fe640 10 monclient(hunting): _negotiate 2024-03-25T19:17:24.069 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.069+0000 7f88077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:24.069 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.069+0000 7f88077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:24.069 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.069+0000 7f88077fe640 10 monclient(hunting): my global_id is 4369 2024-03-25T19:17:24.069 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.069+0000 7f88077fe640 1 -- 172.21.15.6:0/541897271 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8808145f70 con 0x7f88080b1b00 2024-03-25T19:17:24.079 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.079+0000 7f88077fe640 1 -- 172.21.15.6:0/541897271 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 697225122 0 0) 0x7f88080a96f0 con 0x7f88080b44b0 2024-03-25T19:17:24.079 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.079+0000 7f88077fe640 10 monclient(hunting): _negotiate 2024-03-25T19:17:24.079 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.079+0000 7f88077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:24.079 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.079+0000 7f88077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:24.079 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.079+0000 7f88077fe640 10 monclient(hunting): my global_id is 4365 2024-03-25T19:17:24.079 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.079+0000 7f88077fe640 1 -- 172.21.15.6:0/541897271 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88080ab0e0 con 0x7f88080b44b0 2024-03-25T19:17:24.084 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.084+0000 7f88077fe640 1 -- 172.21.15.6:0/541897271 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1361407665 0 0) 0x7f8808145f70 con 0x7f88080b1b00 2024-03-25T19:17:24.084 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.084+0000 7f88077fe640 1 -- 172.21.15.6:0/541897271 >> v1:172.21.15.6:6790/0 conn(0x7f88080b4e70 legacy=0x7f880813f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:24.084 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.084+0000 7f88077fe640 1 -- 172.21.15.6:0/541897271 >> v1:172.21.15.6:6789/0 conn(0x7f88080b44b0 legacy=0x7f8808145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:24.084 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.084+0000 7f88077fe640 10 monclient: _finish_hunting 0 2024-03-25T19:17:24.084 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.084+0000 7f88077fe640 1 monclient: found mon.noname-c 2024-03-25T19:17:24.085 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.084+0000 7f88077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:17:24.085 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.084+0000 7f88077fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:17:24.085 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.084+0000 7f88077fe640 1 -- 172.21.15.6:0/541897271 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88080a9b80 con 0x7f88080b1b00 2024-03-25T19:17:24.085 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.084+0000 7f88077fe640 10 monclient: _finish_auth 0 2024-03-25T19:17:24.085 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.084+0000 7f88077fe640 10 monclient: _check_auth_tickets 2024-03-25T19:17:24.085 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.084+0000 7f88077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:17:24.085 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.084+0000 7f88077fe640 1 -- 172.21.15.6:0/541897271 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f88000019f0 con 0x7f88080b1b00 2024-03-25T19:17:24.085 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.084+0000 7f880ee86640 5 monclient: authenticate success, global_id 4369 2024-03-25T19:17:24.085 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.084+0000 7f88077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:17:24.085 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.084+0000 7f88077fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:17:24.085 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.084+0000 7f88077fe640 10 monclient: dump: 2024-03-25T19:17:24.085 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:17:24.085 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:24.085 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:24.085 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:24.086 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:17:24.086 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:24.086 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:17:24.086 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:17:24.086 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:17:24.086 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:24.086 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.084+0000 7f880ee86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:17:24.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.104+0000 7f88077fe640 1 -- 172.21.15.6:0/541897271 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8800001b50 con 0x7f88080b1b00 2024-03-25T19:17:24.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.104+0000 7f88077fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:17:24.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.104+0000 7f88077fe640 1 -- 172.21.15.6:0/541897271 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f8800001f60 con 0x7f88080b1b00 2024-03-25T19:17:24.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.104+0000 7f880ee86640 10 monclient: get_monmap_and_config success 2024-03-25T19:17:24.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.104+0000 7f880ee86640 4 set_mon_vals no callback set 2024-03-25T19:17:24.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.104+0000 7f88077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:17:24.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.104+0000 7f88077fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:17:24.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.104+0000 7f88077fe640 10 monclient: dump: 2024-03-25T19:17:24.104 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:17:24.105 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:24.105 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:24.105 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:24.105 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:17:24.105 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:24.105 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:17:24.105 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:17:24.105 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:17:24.105 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:24.106 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.104+0000 7f880ee86640 10 monclient: shutdown 2024-03-25T19:17:24.107 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.104+0000 7f880ee86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:17:24.107 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.104+0000 7f880ee86640 1 -- 172.21.15.6:0/541897271 >> v1:172.21.15.192:6789/0 conn(0x7f88080b1b00 legacy=0x7f880814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:24.107 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.105+0000 7f880ee86640 1 -- 172.21.15.6:0/541897271 shutdown_connections 2024-03-25T19:17:24.107 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.105+0000 7f880ee86640 1 -- 172.21.15.6:0/541897271 >> 172.21.15.6:0/541897271 conn(0x7f88080617a0 msgr2=0x7f8808061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:17:24.107 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.105+0000 7f880ee86640 1 -- 172.21.15.6:0/541897271 shutdown_connections 2024-03-25T19:17:24.107 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.105+0000 7f880ee86640 1 -- 172.21.15.6:0/541897271 wait complete. 2024-03-25T19:17:24.107 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.105+0000 7f880ee86640 10 monclient: build_initial_monmap 2024-03-25T19:17:24.107 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.105+0000 7f880ee86640 10 monclient: monmap: 2024-03-25T19:17:24.107 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:17:24.107 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:17:24.107 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:17:24.107 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:17:24.107 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:17:24.108 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:17:24.108 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:17:24.108 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:17:24.108 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:17:24.108 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:24.108 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.106+0000 7f880ee86640 1 librados: starting msgr at 2024-03-25T19:17:24.108 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.106+0000 7f880ee86640 1 librados: starting objecter 2024-03-25T19:17:24.108 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.106+0000 7f880ee86640 1 Processor -- start 2024-03-25T19:17:24.108 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.106+0000 7f880ee86640 1 -- start start 2024-03-25T19:17:24.108 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.106+0000 7f880ee86640 1 librados: setting wanted keys 2024-03-25T19:17:24.108 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.106+0000 7f880ee86640 1 librados: calling monclient init 2024-03-25T19:17:24.108 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.106+0000 7f880ee86640 10 monclient: init 2024-03-25T19:17:24.109 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.107+0000 7f880ee86640 10 monclient: _reopen_session rank -1 2024-03-25T19:17:24.109 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.107+0000 7f880ee86640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:17:24.109 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.107+0000 7f880ee86640 10 monclient(hunting): picked mon.noname-b con 0x7f88080b1b00 addr v1:172.21.15.192:6789/0 2024-03-25T19:17:24.109 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.107+0000 7f880ee86640 10 monclient(hunting): picked mon.noname-c con 0x7f88080b44b0 addr v1:172.21.15.6:6790/0 2024-03-25T19:17:24.110 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.107+0000 7f880ee86640 10 monclient(hunting): picked mon.noname-a con 0x7f88080b4e70 addr v1:172.21.15.6:6789/0 2024-03-25T19:17:24.110 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.107+0000 7f880ee86640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8808136a20 con 0x7f88080b4e70 2024-03-25T19:17:24.110 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.107+0000 7f880ee86640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8808137d20 con 0x7f88080b1b00 2024-03-25T19:17:24.110 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.107+0000 7f880ee86640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88081d7b70 con 0x7f88080b44b0 2024-03-25T19:17:24.110 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.107+0000 7f880ee86640 10 monclient(hunting): _renew_subs 2024-03-25T19:17:24.110 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.107+0000 7f880ee86640 10 monclient(hunting): authenticate will time out at 6148.347475s 2024-03-25T19:17:24.123 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.123+0000 7f8807fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f88080b44b0 0x7f88081d8090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:35270/0 (socket says 172.21.15.6:35270) 2024-03-25T19:17:24.123 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.123+0000 7f8807fff640 1 -- 172.21.15.6:0/1061954324 learned_addr learned my addr 172.21.15.6:0/1061954324 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:17:24.123 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.124+0000 7f88057fa640 10 client.?.objecter ms_handle_connect 0x7f88080b44b0 2024-03-25T19:17:24.123 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.124+0000 7f88057fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:17:24.128 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.128+0000 7f88057fa640 10 client.?.objecter ms_handle_connect 0x7f88080b1b00 2024-03-25T19:17:24.128 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.128+0000 7f88057fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:17:24.131 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.131+0000 7f88057fa640 10 client.?.objecter ms_handle_connect 0x7f88080b4e70 2024-03-25T19:17:24.131 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.131+0000 7f88057fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:17:24.138 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.138+0000 7f88057fa640 1 -- 172.21.15.6:0/1061954324 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3614564460 0 0) 0x7f88081d7b70 con 0x7f88080b44b0 2024-03-25T19:17:24.138 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.138+0000 7f88057fa640 10 monclient(hunting): _negotiate 2024-03-25T19:17:24.138 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.138+0000 7f88057fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:24.138 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.138+0000 7f88057fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:24.138 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.138+0000 7f88057fa640 10 monclient(hunting): my global_id is 4370 2024-03-25T19:17:24.138 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.138+0000 7f88057fa640 1 -- 172.21.15.6:0/1061954324 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87ec004eb0 con 0x7f88080b44b0 2024-03-25T19:17:24.140 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.141+0000 7f88057fa640 1 -- 172.21.15.6:0/1061954324 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2409693407 0 0) 0x7f8808137d20 con 0x7f88080b1b00 2024-03-25T19:17:24.141 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.141+0000 7f88057fa640 10 monclient(hunting): _negotiate 2024-03-25T19:17:24.141 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.141+0000 7f88057fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:24.141 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.141+0000 7f88057fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:24.141 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.141+0000 7f88057fa640 10 monclient(hunting): my global_id is 4372 2024-03-25T19:17:24.141 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.141+0000 7f88057fa640 1 -- 172.21.15.6:0/1061954324 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88081d7b70 con 0x7f88080b1b00 2024-03-25T19:17:24.146 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.146+0000 7f88057fa640 1 -- 172.21.15.6:0/1061954324 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4159854099 0 0) 0x7f8808136a20 con 0x7f88080b4e70 2024-03-25T19:17:24.146 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.146+0000 7f88057fa640 10 monclient(hunting): _negotiate 2024-03-25T19:17:24.147 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.146+0000 7f88057fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:24.147 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.146+0000 7f88057fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:24.147 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.146+0000 7f88057fa640 10 monclient(hunting): my global_id is 4368 2024-03-25T19:17:24.147 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.146+0000 7f88057fa640 1 -- 172.21.15.6:0/1061954324 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8808137d20 con 0x7f88080b4e70 2024-03-25T19:17:24.156 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.155+0000 7f88057fa640 1 -- 172.21.15.6:0/1061954324 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 789343978 0 0) 0x7f88081d7b70 con 0x7f88080b1b00 2024-03-25T19:17:24.156 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.155+0000 7f88057fa640 1 -- 172.21.15.6:0/1061954324 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8808136a20 con 0x7f88080b1b00 2024-03-25T19:17:24.156 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.155+0000 7f88057fa640 1 -- 172.21.15.6:0/1061954324 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f87fc001770 con 0x7f88080b1b00 2024-03-25T19:17:24.156 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.155+0000 7f88057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:17:24.156 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.155+0000 7f88057fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:17:24.156 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.156+0000 7f88057fa640 10 monclient(hunting): dump: 2024-03-25T19:17:24.156 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:17:24.156 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:24.156 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:24.156 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:24.156 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:17:24.156 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:24.156 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:17:24.157 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:17:24.157 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:17:24.157 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:24.157 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.156+0000 7f88057fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:17:24.166 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.166+0000 7f88057fa640 1 -- 172.21.15.6:0/1061954324 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2388882568 0 0) 0x7f8808137d20 con 0x7f88080b4e70 2024-03-25T19:17:24.166 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.166+0000 7f88057fa640 1 -- 172.21.15.6:0/1061954324 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88081d7b70 con 0x7f88080b4e70 2024-03-25T19:17:24.166 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.166+0000 7f88057fa640 1 -- 172.21.15.6:0/1061954324 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f8800001e60 con 0x7f88080b4e70 2024-03-25T19:17:24.166 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.166+0000 7f88057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:17:24.166 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.166+0000 7f88057fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:17:24.166 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.166+0000 7f88057fa640 10 monclient(hunting): dump: 2024-03-25T19:17:24.167 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:17:24.167 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:24.167 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:24.167 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:24.167 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:17:24.167 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:24.167 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:17:24.167 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:17:24.167 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:17:24.167 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:24.168 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.168+0000 7f88057fa640 1 -- 172.21.15.6:0/1061954324 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1655972099 0 0) 0x7f87ec004eb0 con 0x7f88080b44b0 2024-03-25T19:17:24.168 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.168+0000 7f88057fa640 1 -- 172.21.15.6:0/1061954324 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8808137d20 con 0x7f88080b44b0 2024-03-25T19:17:24.168 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.168+0000 7f88057fa640 1 -- 172.21.15.6:0/1061954324 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f87f40019d0 con 0x7f88080b44b0 2024-03-25T19:17:24.168 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.168+0000 7f88057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:17:24.168 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.168+0000 7f88057fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:17:24.168 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.168+0000 7f88057fa640 10 monclient(hunting): dump: 2024-03-25T19:17:24.168 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:17:24.168 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:24.168 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:24.169 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:24.169 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:17:24.169 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:24.169 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:17:24.169 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:17:24.169 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:17:24.169 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:24.172 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.171+0000 7f88057fa640 1 -- 172.21.15.6:0/1061954324 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1048163626 0 0) 0x7f8808136a20 con 0x7f88080b1b00 2024-03-25T19:17:24.172 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.171+0000 7f88057fa640 1 -- 172.21.15.6:0/1061954324 >> v1:172.21.15.6:6790/0 conn(0x7f88080b44b0 legacy=0x7f88081d8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:24.172 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.171+0000 7f88057fa640 1 -- 172.21.15.6:0/1061954324 >> v1:172.21.15.6:6789/0 conn(0x7f88080b4e70 legacy=0x7f88081d4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:24.172 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.171+0000 7f88057fa640 10 monclient: _finish_hunting 0 2024-03-25T19:17:24.172 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.171+0000 7f88057fa640 1 monclient: found mon.b 2024-03-25T19:17:24.173 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.171+0000 7f88057fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:17:24.173 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.171+0000 7f88057fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:17:24.173 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.171+0000 7f88057fa640 1 -- 172.21.15.6:0/1061954324 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88081d87a0 con 0x7f88080b1b00 2024-03-25T19:17:24.173 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.171+0000 7f88057fa640 10 monclient: _finish_auth 0 2024-03-25T19:17:24.173 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.171+0000 7f88057fa640 10 monclient: _check_auth_tickets 2024-03-25T19:17:24.173 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.171+0000 7f88057fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:17:24.173 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.171+0000 7f880ee86640 5 monclient: authenticate success, global_id 4372 2024-03-25T19:17:24.173 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.171+0000 7f880ee86640 10 monclient: _renew_subs 2024-03-25T19:17:24.173 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.171+0000 7f880ee86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:17:24.173 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.171+0000 7f880ee86640 1 -- 172.21.15.6:0/1061954324 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88081d4540 con 0x7f88080b1b00 2024-03-25T19:17:24.173 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.171+0000 7f880ee86640 10 client.4372.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:17:24.173 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.171+0000 7f880ee86640 10 monclient: _renew_subs 2024-03-25T19:17:24.173 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.171+0000 7f880ee86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:17:24.173 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.171+0000 7f880ee86640 1 -- 172.21.15.6:0/1061954324 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8808136e30 con 0x7f88080b1b00 2024-03-25T19:17:24.173 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.171+0000 7f880ee86640 1 librados: init done 2024-03-25T19:17:24.174 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.172+0000 7f880ee86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:17:24.174 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.172+0000 7f880ee86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:17:24.174 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.172+0000 7f880ee86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:17:24.174 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.172+0000 7f880ee86640 1 -- 172.21.15.6:0/1061954324 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88081d8990 con 0x7f88080b1b00 2024-03-25T19:17:24.448 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.447+0000 7f88057fa640 1 -- 172.21.15.6:0/1061954324 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87fc001ba0 con 0x7f88080b1b00 2024-03-25T19:17:24.448 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.447+0000 7f88057fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:17:24.448 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.447+0000 7f88057fa640 1 -- 172.21.15.6:0/1061954324 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f87fc0047e0 con 0x7f88080b1b00 2024-03-25T19:17:24.448 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.447+0000 7f88057fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:17:24.448 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.448+0000 7f88057fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:17:24.448 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.448+0000 7f88057fa640 10 monclient: dump: 2024-03-25T19:17:24.448 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:17:24.449 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:24.449 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:24.449 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:24.449 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:17:24.449 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:24.449 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:17:24.449 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:17:24.449 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:17:24.449 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:24.449 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.448+0000 7f8806ffd640 4 set_mon_vals no callback set 2024-03-25T19:17:24.450 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.449+0000 7f88057fa640 1 -- 172.21.15.6:0/1061954324 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f87fc01a7b0 con 0x7f88080b1b00 2024-03-25T19:17:24.450 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.449+0000 7f8807fff640 10 monclient: get_auth_request con 0x7f87ec06e0f0 auth_method 0 2024-03-25T19:17:24.450 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.450+0000 7f88057fa640 10 client.4372.objecter ms_handle_connect 0x7f87ec06e0f0 2024-03-25T19:17:24.455 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.455+0000 7f88057fa640 1 -- 172.21.15.6:0/1061954324 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f87fc0544e0 con 0x7f88080b1b00 2024-03-25T19:17:24.455 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.455+0000 7f88057fa640 10 client.4372.objecter ms_dispatch 0x7f8808000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:17:24.456 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.455+0000 7f88057fa640 3 client.4372.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:17:24.456 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.455+0000 7f88057fa640 3 client.4372.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:17:24.456 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.455+0000 7f88057fa640 20 client.4372.objecter dump_active .. 0 homeless 2024-03-25T19:17:24.456 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.456+0000 7f88057fa640 1 -- 172.21.15.6:0/1061954324 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f87fc04fe20 con 0x7f88080b1b00 2024-03-25T19:17:24.456 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.456+0000 7f88057fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:17:24.456 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.456+0000 7f88057fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:17:24.719 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.719+0000 7f880ee86640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:17:24.719 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.719+0000 7f880ee86640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:17:24.719 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.719+0000 7f880ee86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:17:24.719 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.719+0000 7f880ee86640 1 -- 172.21.15.6:0/1061954324 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88081d7d00 con 0x7f88080b1b00 2024-03-25T19:17:24.755 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.755+0000 7f88057fa640 1 -- 172.21.15.6:0/1061954324 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f87fc051f20 con 0x7f88080b1b00 2024-03-25T19:17:24.756 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.755+0000 7f88057fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:17:24.756 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.755+0000 7f88057fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:17:24.756 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:17:24.756 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:17:24.758 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.758+0000 7f880ee86640 10 librados: watch_flush enter 2024-03-25T19:17:24.758 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.758+0000 7f880ee86640 10 librados: watch_flush exit 2024-03-25T19:17:24.758 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.758+0000 7f880ee86640 20 client.4372.objecter shutdown clearing up homeless session... 2024-03-25T19:17:24.759 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.758+0000 7f880ee86640 10 client.4372.objecter successfully canceled tick 2024-03-25T19:17:24.759 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.758+0000 7f880ee86640 1 -- 172.21.15.6:0/1061954324 >> v1:172.21.15.6:6800/2936652650 conn(0x7f87ec06e0f0 legacy=0x7f87ec0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:24.759 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.758+0000 7f880ee86640 10 monclient: shutdown 2024-03-25T19:17:24.759 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.758+0000 7f880ee86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:17:24.759 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.758+0000 7f880ee86640 1 -- 172.21.15.6:0/1061954324 >> v1:172.21.15.192:6789/0 conn(0x7f88080b1b00 legacy=0x7f8808136310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:24.759 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.758+0000 7f880ee86640 1 -- 172.21.15.6:0/1061954324 shutdown_connections 2024-03-25T19:17:24.759 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.758+0000 7f880ee86640 1 -- 172.21.15.6:0/1061954324 >> 172.21.15.6:0/1061954324 conn(0x7f88080617a0 msgr2=0x7f88080b6620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:17:24.759 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.758+0000 7f880ee86640 1 -- 172.21.15.6:0/1061954324 shutdown_connections 2024-03-25T19:17:24.759 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.758+0000 7f880ee86640 1 -- 172.21.15.6:0/1061954324 wait complete. 2024-03-25T19:17:24.759 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:24.758+0000 7f880ee86640 1 librados: shutdown 2024-03-25T19:17:24.769 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:17:31.575 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:17:31.684 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.684+0000 7fee7a3cd640 10 monclient: get_monmap_and_config 2024-03-25T19:17:31.684 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.684+0000 7fee7a3cd640 10 monclient: build_initial_monmap 2024-03-25T19:17:31.684 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.684+0000 7fee7a3cd640 10 monclient: monmap: 2024-03-25T19:17:31.685 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:17:31.685 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:31.685 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:17:31.685784+0000 2024-03-25T19:17:31.685 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:17:31.685784+0000 2024-03-25T19:17:31.685 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:17:31.685 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:31.685 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:17:31.685 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:17:31.685 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:17:31.685 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:31.686 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.685+0000 7fee7a3cd640 1 Processor -- start 2024-03-25T19:17:31.686 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.685+0000 7fee7a3cd640 1 -- start start 2024-03-25T19:17:31.686 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.685+0000 7fee7a3cd640 10 monclient: init 2024-03-25T19:17:31.686 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.686+0000 7fee7a3cd640 10 monclient: _reopen_session rank -1 2024-03-25T19:17:31.687 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.686+0000 7fee7a3cd640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:17:31.687 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.686+0000 7fee7a3cd640 10 monclient(hunting): picked mon.noname-a con 0x7fee740b4ff0 addr v1:172.21.15.6:6789/0 2024-03-25T19:17:31.687 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.686+0000 7fee7a3cd640 10 monclient(hunting): picked mon.noname-b con 0x7fee740b4670 addr v1:172.21.15.6:6790/0 2024-03-25T19:17:31.687 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.686+0000 7fee7a3cd640 10 monclient(hunting): picked mon.noname-c con 0x7fee7413c480 addr v1:172.21.15.192:6789/0 2024-03-25T19:17:31.687 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.686+0000 7fee7a3cd640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee740ab0e0 con 0x7fee740b4ff0 2024-03-25T19:17:31.687 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.686+0000 7fee7a3cd640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee740ab3c0 con 0x7fee7413c480 2024-03-25T19:17:31.687 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.686+0000 7fee7a3cd640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee740a9f30 con 0x7fee740b4670 2024-03-25T19:17:31.687 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.686+0000 7fee7a3cd640 10 monclient(hunting): _renew_subs 2024-03-25T19:17:31.687 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.686+0000 7fee7a3cd640 10 monclient(hunting): authenticate will time out at 6155.926454s 2024-03-25T19:17:31.697 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.697+0000 7fee73fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fee740b4ff0 0x7fee7413bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35778/0 (socket says 172.21.15.6:35778) 2024-03-25T19:17:31.697 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.697+0000 7fee73fff640 1 -- 172.21.15.6:0/3411693423 learned_addr learned my addr 172.21.15.6:0/3411693423 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:17:31.715 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.715+0000 7fee72ffd640 1 -- 172.21.15.6:0/3411693423 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2709041100 0 0) 0x7fee740ab0e0 con 0x7fee740b4ff0 2024-03-25T19:17:31.715 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.715+0000 7fee72ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:17:31.715 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.715+0000 7fee72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:31.715 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.715+0000 7fee72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:31.715 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.715+0000 7fee72ffd640 10 monclient(hunting): my global_id is 4371 2024-03-25T19:17:31.715 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.715+0000 7fee72ffd640 1 -- 172.21.15.6:0/3411693423 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee5c003520 con 0x7fee740b4ff0 2024-03-25T19:17:31.717 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.717+0000 7fee72ffd640 1 -- 172.21.15.6:0/3411693423 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4104757144 0 0) 0x7fee740ab3c0 con 0x7fee7413c480 2024-03-25T19:17:31.717 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.717+0000 7fee72ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:17:31.717 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.717+0000 7fee72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:31.717 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.717+0000 7fee72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:31.717 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.717+0000 7fee72ffd640 10 monclient(hunting): my global_id is 4375 2024-03-25T19:17:31.717 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.717+0000 7fee72ffd640 1 -- 172.21.15.6:0/3411693423 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee740ab0e0 con 0x7fee7413c480 2024-03-25T19:17:31.727 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.727+0000 7fee72ffd640 1 -- 172.21.15.6:0/3411693423 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3757209153 0 0) 0x7fee5c003520 con 0x7fee740b4ff0 2024-03-25T19:17:31.727 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.727+0000 7fee72ffd640 1 -- 172.21.15.6:0/3411693423 >> v1:172.21.15.6:6790/0 conn(0x7fee740b4670 legacy=0x7fee741425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:31.727 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.727+0000 7fee72ffd640 1 -- 172.21.15.6:0/3411693423 >> v1:172.21.15.192:6789/0 conn(0x7fee7413c480 legacy=0x7fee7414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:31.727 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.727+0000 7fee72ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:17:31.727 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.727+0000 7fee72ffd640 1 monclient: found mon.noname-a 2024-03-25T19:17:31.727 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.727+0000 7fee72ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:17:31.727 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.727+0000 7fee72ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-03-25T19:17:31.727 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.727+0000 7fee72ffd640 1 -- 172.21.15.6:0/3411693423 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee740a9b70 con 0x7fee740b4ff0 2024-03-25T19:17:31.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.727+0000 7fee72ffd640 10 monclient: _finish_auth 0 2024-03-25T19:17:31.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.727+0000 7fee72ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:17:31.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.727+0000 7fee72ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:17:31.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.727+0000 7fee72ffd640 1 -- 172.21.15.6:0/3411693423 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fee64001690 con 0x7fee740b4ff0 2024-03-25T19:17:31.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.727+0000 7fee7a3cd640 5 monclient: authenticate success, global_id 4371 2024-03-25T19:17:31.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.727+0000 7fee7a3cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:17:31.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.727+0000 7fee72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:17:31.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.727+0000 7fee72ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:17:31.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.727+0000 7fee72ffd640 10 monclient: dump: 2024-03-25T19:17:31.728 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:17:31.728 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:31.728 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:31.728 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:31.728 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:17:31.729 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:31.729 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:17:31.729 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:17:31.729 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:17:31.729 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:31.729 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.727+0000 7fee7a3cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:17:31.743 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.743+0000 7fee72ffd640 1 -- 172.21.15.6:0/3411693423 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee640017f0 con 0x7fee740b4ff0 2024-03-25T19:17:31.743 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.743+0000 7fee72ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:17:31.743 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.743+0000 7fee72ffd640 1 -- 172.21.15.6:0/3411693423 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fee64001c00 con 0x7fee740b4ff0 2024-03-25T19:17:31.743 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.743+0000 7fee7a3cd640 10 monclient: get_monmap_and_config success 2024-03-25T19:17:31.743 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.743+0000 7fee7a3cd640 4 set_mon_vals no callback set 2024-03-25T19:17:31.744 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.743+0000 7fee72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:17:31.744 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.743+0000 7fee72ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:17:31.744 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.743+0000 7fee72ffd640 10 monclient: dump: 2024-03-25T19:17:31.744 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:17:31.744 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:31.744 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:31.744 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:31.744 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:17:31.744 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:31.744 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:17:31.744 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:17:31.744 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:17:31.744 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:31.746 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.744+0000 7fee7a3cd640 10 monclient: shutdown 2024-03-25T19:17:31.746 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.744+0000 7fee7a3cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:17:31.746 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.744+0000 7fee7a3cd640 1 -- 172.21.15.6:0/3411693423 >> v1:172.21.15.6:6789/0 conn(0x7fee740b4ff0 legacy=0x7fee7413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:31.746 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.744+0000 7fee7a3cd640 1 -- 172.21.15.6:0/3411693423 shutdown_connections 2024-03-25T19:17:31.746 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.744+0000 7fee7a3cd640 1 -- 172.21.15.6:0/3411693423 >> 172.21.15.6:0/3411693423 conn(0x7fee740617a0 msgr2=0x7fee74061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:17:31.746 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.744+0000 7fee7a3cd640 1 -- 172.21.15.6:0/3411693423 shutdown_connections 2024-03-25T19:17:31.746 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.744+0000 7fee7a3cd640 1 -- 172.21.15.6:0/3411693423 wait complete. 2024-03-25T19:17:31.746 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.745+0000 7fee7a3cd640 10 monclient: build_initial_monmap 2024-03-25T19:17:31.746 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.745+0000 7fee7a3cd640 10 monclient: monmap: 2024-03-25T19:17:31.746 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:17:31.746 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:17:31.746 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:17:31.746 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:17:31.747 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:17:31.747 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:17:31.747 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:17:31.747 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:17:31.747 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:17:31.747 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:31.747 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.745+0000 7fee7a3cd640 1 librados: starting msgr at 2024-03-25T19:17:31.747 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.745+0000 7fee7a3cd640 1 librados: starting objecter 2024-03-25T19:17:31.747 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.746+0000 7fee7a3cd640 1 Processor -- start 2024-03-25T19:17:31.747 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.746+0000 7fee7a3cd640 1 -- start start 2024-03-25T19:17:31.747 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.746+0000 7fee7a3cd640 1 librados: setting wanted keys 2024-03-25T19:17:31.747 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.746+0000 7fee7a3cd640 1 librados: calling monclient init 2024-03-25T19:17:31.747 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.746+0000 7fee7a3cd640 10 monclient: init 2024-03-25T19:17:31.748 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.747+0000 7fee7a3cd640 10 monclient: _reopen_session rank -1 2024-03-25T19:17:31.748 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.747+0000 7fee7a3cd640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:17:31.748 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.747+0000 7fee7a3cd640 10 monclient(hunting): picked mon.noname-c con 0x7fee740b4670 addr v1:172.21.15.6:6790/0 2024-03-25T19:17:31.749 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.747+0000 7fee7a3cd640 10 monclient(hunting): picked mon.noname-b con 0x7fee740b4ff0 addr v1:172.21.15.192:6789/0 2024-03-25T19:17:31.749 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.747+0000 7fee7a3cd640 10 monclient(hunting): picked mon.noname-a con 0x7fee7413c480 addr v1:172.21.15.6:6789/0 2024-03-25T19:17:31.749 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.747+0000 7fee7a3cd640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee74135ea0 con 0x7fee7413c480 2024-03-25T19:17:31.749 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.747+0000 7fee7a3cd640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee740ac3d0 con 0x7fee740b4ff0 2024-03-25T19:17:31.749 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.747+0000 7fee7a3cd640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee740ac6b0 con 0x7fee740b4670 2024-03-25T19:17:31.749 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.747+0000 7fee7a3cd640 10 monclient(hunting): _renew_subs 2024-03-25T19:17:31.749 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.747+0000 7fee7a3cd640 10 monclient(hunting): authenticate will time out at 6155.986933s 2024-03-25T19:17:31.763 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.763+0000 7fee73fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fee740b4670 0x7fee7413b8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:47376/0 (socket says 172.21.15.6:47376) 2024-03-25T19:17:31.763 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.763+0000 7fee73fff640 1 -- 172.21.15.6:0/1725380270 learned_addr learned my addr 172.21.15.6:0/1725380270 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:17:31.768 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.768+0000 7fee70ff9640 10 client.?.objecter ms_handle_connect 0x7fee740b4ff0 2024-03-25T19:17:31.768 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.768+0000 7fee70ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:17:31.770 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.770+0000 7fee70ff9640 10 client.?.objecter ms_handle_connect 0x7fee7413c480 2024-03-25T19:17:31.770 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.770+0000 7fee70ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:17:31.771 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.772+0000 7fee70ff9640 10 client.?.objecter ms_handle_connect 0x7fee740b4670 2024-03-25T19:17:31.771 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.772+0000 7fee70ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:17:31.779 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.779+0000 7fee70ff9640 1 -- 172.21.15.6:0/1725380270 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1706287583 0 0) 0x7fee74135ea0 con 0x7fee7413c480 2024-03-25T19:17:31.779 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.779+0000 7fee70ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:17:31.780 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.779+0000 7fee70ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:31.780 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.779+0000 7fee70ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:31.780 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.779+0000 7fee70ff9640 10 monclient(hunting): my global_id is 4374 2024-03-25T19:17:31.780 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.779+0000 7fee70ff9640 1 -- 172.21.15.6:0/1725380270 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee60004eb0 con 0x7fee7413c480 2024-03-25T19:17:31.782 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.782+0000 7fee70ff9640 1 -- 172.21.15.6:0/1725380270 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3592142094 0 0) 0x7fee740ac3d0 con 0x7fee740b4ff0 2024-03-25T19:17:31.782 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.782+0000 7fee70ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:17:31.782 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.782+0000 7fee70ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:31.782 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.782+0000 7fee70ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:31.782 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.782+0000 7fee70ff9640 10 monclient(hunting): my global_id is 4378 2024-03-25T19:17:31.782 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.782+0000 7fee70ff9640 1 -- 172.21.15.6:0/1725380270 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee74135ea0 con 0x7fee740b4ff0 2024-03-25T19:17:31.786 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.786+0000 7fee70ff9640 1 -- 172.21.15.6:0/1725380270 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1719661493 0 0) 0x7fee740ac6b0 con 0x7fee740b4670 2024-03-25T19:17:31.786 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.786+0000 7fee70ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:17:31.786 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.786+0000 7fee70ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:31.786 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.786+0000 7fee70ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:31.786 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.786+0000 7fee70ff9640 10 monclient(hunting): my global_id is 4376 2024-03-25T19:17:31.786 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.786+0000 7fee70ff9640 1 -- 172.21.15.6:0/1725380270 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee740ac3d0 con 0x7fee740b4670 2024-03-25T19:17:31.792 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.792+0000 7fee70ff9640 1 -- 172.21.15.6:0/1725380270 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4070031566 0 0) 0x7fee60004eb0 con 0x7fee7413c480 2024-03-25T19:17:31.792 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.792+0000 7fee70ff9640 1 -- 172.21.15.6:0/1725380270 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee740ac6b0 con 0x7fee7413c480 2024-03-25T19:17:31.792 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.792+0000 7fee70ff9640 1 -- 172.21.15.6:0/1725380270 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fee6c001bd0 con 0x7fee7413c480 2024-03-25T19:17:31.792 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.792+0000 7fee70ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:17:31.792 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.792+0000 7fee70ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:17:31.792 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.792+0000 7fee70ff9640 10 monclient(hunting): dump: 2024-03-25T19:17:31.792 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:17:31.793 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:31.793 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:31.793 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:31.793 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:17:31.793 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:31.793 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:17:31.793 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:17:31.793 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:17:31.793 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:31.793 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.792+0000 7fee70ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:17:31.797 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.797+0000 7fee70ff9640 1 -- 172.21.15.6:0/1725380270 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 734129129 0 0) 0x7fee740ac3d0 con 0x7fee740b4670 2024-03-25T19:17:31.797 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.797+0000 7fee70ff9640 1 -- 172.21.15.6:0/1725380270 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee60004eb0 con 0x7fee740b4670 2024-03-25T19:17:31.797 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.797+0000 7fee70ff9640 1 -- 172.21.15.6:0/1725380270 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fee64001bc0 con 0x7fee740b4670 2024-03-25T19:17:31.797 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.797+0000 7fee70ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:17:31.798 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.797+0000 7fee70ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:17:31.798 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.797+0000 7fee70ff9640 10 monclient(hunting): dump: 2024-03-25T19:17:31.798 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:17:31.798 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:31.798 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:31.798 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:31.798 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:17:31.798 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:31.798 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:17:31.798 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:17:31.798 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:17:31.798 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:31.801 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.801+0000 7fee70ff9640 1 -- 172.21.15.6:0/1725380270 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3791639583 0 0) 0x7fee740ac6b0 con 0x7fee7413c480 2024-03-25T19:17:31.801 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.801+0000 7fee70ff9640 1 -- 172.21.15.6:0/1725380270 >> v1:172.21.15.6:6790/0 conn(0x7fee740b4670 legacy=0x7fee7413b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:31.801 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.801+0000 7fee70ff9640 1 -- 172.21.15.6:0/1725380270 >> v1:172.21.15.192:6789/0 conn(0x7fee740b4ff0 legacy=0x7fee740b0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:31.801 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.801+0000 7fee70ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:17:31.801 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.801+0000 7fee70ff9640 1 monclient: found mon.a 2024-03-25T19:17:31.801 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.801+0000 7fee70ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:17:31.802 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.801+0000 7fee70ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:17:31.802 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.801+0000 7fee70ff9640 1 -- 172.21.15.6:0/1725380270 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee7413c110 con 0x7fee7413c480 2024-03-25T19:17:31.802 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.801+0000 7fee70ff9640 10 monclient: _finish_auth 0 2024-03-25T19:17:31.802 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.801+0000 7fee70ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:17:31.802 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.801+0000 7fee70ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:17:31.802 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.801+0000 7fee7a3cd640 5 monclient: authenticate success, global_id 4374 2024-03-25T19:17:31.802 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.801+0000 7fee7a3cd640 10 monclient: _renew_subs 2024-03-25T19:17:31.802 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.801+0000 7fee7a3cd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:17:31.802 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.801+0000 7fee7a3cd640 1 -- 172.21.15.6:0/1725380270 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee7413ae10 con 0x7fee7413c480 2024-03-25T19:17:31.802 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.801+0000 7fee7a3cd640 10 client.4374.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:17:31.802 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.801+0000 7fee7a3cd640 10 monclient: _renew_subs 2024-03-25T19:17:31.802 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.801+0000 7fee7a3cd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:17:31.802 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.801+0000 7fee7a3cd640 1 -- 172.21.15.6:0/1725380270 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fee740b1730 con 0x7fee7413c480 2024-03-25T19:17:31.802 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.801+0000 7fee7a3cd640 1 librados: init done 2024-03-25T19:17:31.803 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.802+0000 7fee7a3cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:17:31.803 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.802+0000 7fee7a3cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:17:31.803 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.802+0000 7fee7a3cd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:17:31.803 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:31.802+0000 7fee7a3cd640 1 -- 172.21.15.6:0/1725380270 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee7413b1b0 con 0x7fee7413c480 2024-03-25T19:17:32.123 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:32.122+0000 7fee70ff9640 1 -- 172.21.15.6:0/1725380270 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee6c0020c0 con 0x7fee7413c480 2024-03-25T19:17:32.123 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:32.122+0000 7fee70ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:17:32.123 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:32.122+0000 7fee70ff9640 1 -- 172.21.15.6:0/1725380270 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fee6c003b30 con 0x7fee7413c480 2024-03-25T19:17:32.123 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:32.122+0000 7fee70ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:17:32.123 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:32.122+0000 7fee70ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:17:32.123 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:32.122+0000 7fee70ff9640 10 monclient: dump: 2024-03-25T19:17:32.123 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:17:32.123 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:32.123 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:32.123 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:32.124 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:17:32.124 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:32.124 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:17:32.124 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:17:32.124 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:17:32.124 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:32.124 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:32.122+0000 7fee727fc640 4 set_mon_vals no callback set 2024-03-25T19:17:32.124 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:32.123+0000 7fee70ff9640 1 -- 172.21.15.6:0/1725380270 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7fee6c019b80 con 0x7fee7413c480 2024-03-25T19:17:32.124 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:32.123+0000 7fee73fff640 10 monclient: get_auth_request con 0x7fee6006df20 auth_method 0 2024-03-25T19:17:32.125 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:32.124+0000 7fee70ff9640 10 client.4374.objecter ms_handle_connect 0x7fee6006df20 2024-03-25T19:17:32.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:32.129+0000 7fee70ff9640 1 -- 172.21.15.6:0/1725380270 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7fee6c053860 con 0x7fee7413c480 2024-03-25T19:17:32.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:32.129+0000 7fee70ff9640 10 client.4374.objecter ms_dispatch 0x7fee74000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:17:32.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:32.129+0000 7fee70ff9640 3 client.4374.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:17:32.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:32.129+0000 7fee70ff9640 3 client.4374.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:17:32.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:32.129+0000 7fee70ff9640 20 client.4374.objecter dump_active .. 0 homeless 2024-03-25T19:17:32.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:32.129+0000 7fee70ff9640 1 -- 172.21.15.6:0/1725380270 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fee6c04f1a0 con 0x7fee7413c480 2024-03-25T19:17:32.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:32.129+0000 7fee70ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:17:32.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:32.129+0000 7fee70ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:17:32.386 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:32.386+0000 7fee7a3cd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:17:32.386 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:32.386+0000 7fee7a3cd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:17:32.386 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:32.386+0000 7fee7a3cd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:17:32.387 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:32.386+0000 7fee7a3cd640 1 -- 172.21.15.6:0/1725380270 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee7406dbd0 con 0x7fee7413c480 2024-03-25T19:17:32.421 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:32.418+0000 7fee70ff9640 1 -- 172.21.15.6:0/1725380270 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7fee6c0512a0 con 0x7fee7413c480 2024-03-25T19:17:32.422 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:32.418+0000 7fee70ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:17:32.422 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:32.418+0000 7fee70ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:17:32.422 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:17:32.422 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:17:32.423 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:32.420+0000 7fee7a3cd640 10 librados: watch_flush enter 2024-03-25T19:17:32.424 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:32.420+0000 7fee7a3cd640 10 librados: watch_flush exit 2024-03-25T19:17:32.424 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:32.420+0000 7fee7a3cd640 20 client.4374.objecter shutdown clearing up homeless session... 2024-03-25T19:17:32.424 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:32.420+0000 7fee7a3cd640 10 client.4374.objecter successfully canceled tick 2024-03-25T19:17:32.424 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:32.420+0000 7fee7a3cd640 1 -- 172.21.15.6:0/1725380270 >> v1:172.21.15.6:6800/2936652650 conn(0x7fee6006df20 legacy=0x7fee600703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:32.424 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:32.420+0000 7fee7a3cd640 10 monclient: shutdown 2024-03-25T19:17:32.424 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:32.420+0000 7fee7a3cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:17:32.424 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:32.420+0000 7fee7a3cd640 1 -- 172.21.15.6:0/1725380270 >> v1:172.21.15.6:6789/0 conn(0x7fee7413c480 legacy=0x7fee741362d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:32.424 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:32.421+0000 7fee7a3cd640 1 -- 172.21.15.6:0/1725380270 shutdown_connections 2024-03-25T19:17:32.424 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:32.421+0000 7fee7a3cd640 1 -- 172.21.15.6:0/1725380270 >> 172.21.15.6:0/1725380270 conn(0x7fee740617a0 msgr2=0x7fee741407a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:17:32.424 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:32.421+0000 7fee7a3cd640 1 -- 172.21.15.6:0/1725380270 shutdown_connections 2024-03-25T19:17:32.424 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:32.421+0000 7fee7a3cd640 1 -- 172.21.15.6:0/1725380270 wait complete. 2024-03-25T19:17:32.424 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:32.421+0000 7fee7a3cd640 1 librados: shutdown 2024-03-25T19:17:32.433 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:17:39.238 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:17:39.352 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.352+0000 7f9eeb057640 10 monclient: get_monmap_and_config 2024-03-25T19:17:39.352 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.352+0000 7f9eeb057640 10 monclient: build_initial_monmap 2024-03-25T19:17:39.352 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.352+0000 7f9eeb057640 10 monclient: monmap: 2024-03-25T19:17:39.352 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:17:39.352 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:39.352 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:17:39.353470+0000 2024-03-25T19:17:39.353 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:17:39.353470+0000 2024-03-25T19:17:39.353 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:17:39.353 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:39.353 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:17:39.353 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:17:39.353 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:17:39.353 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:39.354 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.353+0000 7f9eeb057640 1 Processor -- start 2024-03-25T19:17:39.354 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.353+0000 7f9eeb057640 1 -- start start 2024-03-25T19:17:39.354 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.353+0000 7f9eeb057640 10 monclient: init 2024-03-25T19:17:39.354 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.354+0000 7f9eeb057640 10 monclient: _reopen_session rank -1 2024-03-25T19:17:39.355 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.354+0000 7f9eeb057640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:17:39.355 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.354+0000 7f9eeb057640 10 monclient(hunting): picked mon.noname-c con 0x7f9ee40b4ff0 addr v1:172.21.15.192:6789/0 2024-03-25T19:17:39.355 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.354+0000 7f9eeb057640 10 monclient(hunting): picked mon.noname-b con 0x7f9ee40b4670 addr v1:172.21.15.6:6790/0 2024-03-25T19:17:39.355 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.354+0000 7f9eeb057640 10 monclient(hunting): picked mon.noname-a con 0x7f9ee413c480 addr v1:172.21.15.6:6789/0 2024-03-25T19:17:39.355 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.354+0000 7f9eeb057640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ee40ab0e0 con 0x7f9ee413c480 2024-03-25T19:17:39.355 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.354+0000 7f9eeb057640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ee40ab3c0 con 0x7f9ee40b4ff0 2024-03-25T19:17:39.355 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.354+0000 7f9eeb057640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ee40a9f30 con 0x7f9ee40b4670 2024-03-25T19:17:39.355 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.354+0000 7f9eeb057640 10 monclient(hunting): _renew_subs 2024-03-25T19:17:39.355 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.354+0000 7f9eeb057640 10 monclient(hunting): authenticate will time out at 6163.594009s 2024-03-25T19:17:39.360 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.360+0000 7f9ee8df3640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f9ee40b4ff0 0x7f9ee413bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:50214/0 (socket says 172.21.15.6:50214) 2024-03-25T19:17:39.360 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.360+0000 7f9ee8df3640 1 -- 172.21.15.6:0/3724040362 learned_addr learned my addr 172.21.15.6:0/3724040362 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:17:39.377 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.377+0000 7f9ee37fe640 1 -- 172.21.15.6:0/3724040362 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3637816387 0 0) 0x7f9ee40ab3c0 con 0x7f9ee40b4ff0 2024-03-25T19:17:39.377 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.377+0000 7f9ee37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:17:39.377 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.377+0000 7f9ee37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:39.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.377+0000 7f9ee37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:39.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.377+0000 7f9ee37fe640 10 monclient(hunting): my global_id is 4381 2024-03-25T19:17:39.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.377+0000 7f9ee37fe640 1 -- 172.21.15.6:0/3724040362 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ed4003520 con 0x7f9ee40b4ff0 2024-03-25T19:17:39.388 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.388+0000 7f9ee37fe640 1 -- 172.21.15.6:0/3724040362 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1932206336 0 0) 0x7f9ee40ab0e0 con 0x7f9ee413c480 2024-03-25T19:17:39.388 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.388+0000 7f9ee37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:17:39.388 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.388+0000 7f9ee37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:39.388 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.388+0000 7f9ee37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:39.388 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.388+0000 7f9ee37fe640 10 monclient(hunting): my global_id is 4377 2024-03-25T19:17:39.388 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.388+0000 7f9ee37fe640 1 -- 172.21.15.6:0/3724040362 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ee40ab3c0 con 0x7f9ee413c480 2024-03-25T19:17:39.392 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.392+0000 7f9ee37fe640 1 -- 172.21.15.6:0/3724040362 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1988150193 0 0) 0x7f9ee40a9f30 con 0x7f9ee40b4670 2024-03-25T19:17:39.392 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.392+0000 7f9ee37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:17:39.392 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.392+0000 7f9ee37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:39.392 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.392+0000 7f9ee37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:39.392 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.392+0000 7f9ee37fe640 10 monclient(hunting): my global_id is 4379 2024-03-25T19:17:39.392 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.392+0000 7f9ee37fe640 1 -- 172.21.15.6:0/3724040362 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ee40ab0e0 con 0x7f9ee40b4670 2024-03-25T19:17:39.407 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.407+0000 7f9ee37fe640 1 -- 172.21.15.6:0/3724040362 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 241276000 0 0) 0x7f9ed4003520 con 0x7f9ee40b4ff0 2024-03-25T19:17:39.407 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.407+0000 7f9ee37fe640 1 -- 172.21.15.6:0/3724040362 >> v1:172.21.15.6:6790/0 conn(0x7f9ee40b4670 legacy=0x7f9ee41425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:39.407 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.407+0000 7f9ee37fe640 1 -- 172.21.15.6:0/3724040362 >> v1:172.21.15.6:6789/0 conn(0x7f9ee413c480 legacy=0x7f9ee414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:39.407 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.407+0000 7f9ee37fe640 10 monclient: _finish_hunting 0 2024-03-25T19:17:39.407 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.407+0000 7f9ee37fe640 1 monclient: found mon.noname-c 2024-03-25T19:17:39.407 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.407+0000 7f9ee37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:17:39.407 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.407+0000 7f9ee37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:17:39.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.407+0000 7f9ee37fe640 1 -- 172.21.15.6:0/3724040362 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee40a9b70 con 0x7f9ee40b4ff0 2024-03-25T19:17:39.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.407+0000 7f9ee37fe640 10 monclient: _finish_auth 0 2024-03-25T19:17:39.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.407+0000 7f9ee37fe640 10 monclient: _check_auth_tickets 2024-03-25T19:17:39.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.407+0000 7f9ee37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:17:39.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.407+0000 7f9ee37fe640 1 -- 172.21.15.6:0/3724040362 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f9ed8001690 con 0x7f9ee40b4ff0 2024-03-25T19:17:39.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.407+0000 7f9ee37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:17:39.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.407+0000 7f9ee37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:17:39.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.407+0000 7f9ee37fe640 10 monclient: dump: 2024-03-25T19:17:39.408 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:17:39.408 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:39.408 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:39.408 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:39.408 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:17:39.408 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:39.408 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:17:39.409 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:17:39.409 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:17:39.409 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:39.409 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.407+0000 7f9eeb057640 5 monclient: authenticate success, global_id 4381 2024-03-25T19:17:39.409 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.407+0000 7f9eeb057640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:17:39.437 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.437+0000 7f9ee37fe640 1 -- 172.21.15.6:0/3724040362 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ed80017f0 con 0x7f9ee40b4ff0 2024-03-25T19:17:39.437 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.437+0000 7f9ee37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:17:39.437 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.437+0000 7f9ee37fe640 1 -- 172.21.15.6:0/3724040362 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f9ed8001c00 con 0x7f9ee40b4ff0 2024-03-25T19:17:39.437 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.437+0000 7f9ee37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:17:39.437 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.437+0000 7f9ee37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:17:39.437 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.437+0000 7f9ee37fe640 10 monclient: dump: 2024-03-25T19:17:39.437 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:17:39.437 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:39.437 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:39.438 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:39.438 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:17:39.438 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:39.438 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:17:39.438 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:17:39.438 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:17:39.438 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:39.438 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.437+0000 7f9eeb057640 10 monclient: get_monmap_and_config success 2024-03-25T19:17:39.438 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.437+0000 7f9eeb057640 4 set_mon_vals no callback set 2024-03-25T19:17:39.439 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.438+0000 7f9eeb057640 10 monclient: shutdown 2024-03-25T19:17:39.439 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.438+0000 7f9eeb057640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:17:39.439 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.438+0000 7f9eeb057640 1 -- 172.21.15.6:0/3724040362 >> v1:172.21.15.192:6789/0 conn(0x7f9ee40b4ff0 legacy=0x7f9ee413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:39.440 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.438+0000 7f9eeb057640 1 -- 172.21.15.6:0/3724040362 shutdown_connections 2024-03-25T19:17:39.440 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.438+0000 7f9eeb057640 1 -- 172.21.15.6:0/3724040362 >> 172.21.15.6:0/3724040362 conn(0x7f9ee40617a0 msgr2=0x7f9ee4061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:17:39.440 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.438+0000 7f9eeb057640 1 -- 172.21.15.6:0/3724040362 shutdown_connections 2024-03-25T19:17:39.440 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.438+0000 7f9eeb057640 1 -- 172.21.15.6:0/3724040362 wait complete. 2024-03-25T19:17:39.440 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.439+0000 7f9eeb057640 10 monclient: build_initial_monmap 2024-03-25T19:17:39.440 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.439+0000 7f9eeb057640 10 monclient: monmap: 2024-03-25T19:17:39.440 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:17:39.440 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:17:39.440 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:17:39.440 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:17:39.440 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:17:39.440 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:17:39.440 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:17:39.440 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:17:39.440 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:17:39.440 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:39.441 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.439+0000 7f9eeb057640 1 librados: starting msgr at 2024-03-25T19:17:39.441 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.439+0000 7f9eeb057640 1 librados: starting objecter 2024-03-25T19:17:39.441 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.439+0000 7f9eeb057640 1 Processor -- start 2024-03-25T19:17:39.441 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.439+0000 7f9eeb057640 1 -- start start 2024-03-25T19:17:39.441 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.439+0000 7f9eeb057640 1 librados: setting wanted keys 2024-03-25T19:17:39.441 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.439+0000 7f9eeb057640 1 librados: calling monclient init 2024-03-25T19:17:39.441 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.439+0000 7f9eeb057640 10 monclient: init 2024-03-25T19:17:39.442 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.440+0000 7f9eeb057640 10 monclient: _reopen_session rank -1 2024-03-25T19:17:39.442 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.440+0000 7f9eeb057640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:17:39.442 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.440+0000 7f9eeb057640 10 monclient(hunting): picked mon.noname-c con 0x7f9ee40b4670 addr v1:172.21.15.6:6790/0 2024-03-25T19:17:39.442 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.440+0000 7f9eeb057640 10 monclient(hunting): picked mon.noname-b con 0x7f9ee40b4ff0 addr v1:172.21.15.192:6789/0 2024-03-25T19:17:39.442 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.440+0000 7f9eeb057640 10 monclient(hunting): picked mon.noname-a con 0x7f9ee413c480 addr v1:172.21.15.6:6789/0 2024-03-25T19:17:39.442 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.440+0000 7f9eeb057640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ee4143c80 con 0x7f9ee413c480 2024-03-25T19:17:39.442 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.440+0000 7f9eeb057640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ee4146570 con 0x7f9ee40b4ff0 2024-03-25T19:17:39.442 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.440+0000 7f9eeb057640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ee4130cc0 con 0x7f9ee40b4670 2024-03-25T19:17:39.442 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.440+0000 7f9eeb057640 10 monclient(hunting): _renew_subs 2024-03-25T19:17:39.442 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.440+0000 7f9eeb057640 10 monclient(hunting): authenticate will time out at 6163.680355s 2024-03-25T19:17:39.455 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.456+0000 7f9ee3fff640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f9ee40b4ff0 0x7f9ee4145e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:50272/0 (socket says 172.21.15.6:50272) 2024-03-25T19:17:39.456 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.456+0000 7f9ee3fff640 1 -- 172.21.15.6:0/707600607 learned_addr learned my addr 172.21.15.6:0/707600607 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:17:39.456 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.457+0000 7f9ee17fa640 10 client.?.objecter ms_handle_connect 0x7f9ee40b4ff0 2024-03-25T19:17:39.456 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.457+0000 7f9ee17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:17:39.464 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.464+0000 7f9ee17fa640 10 client.?.objecter ms_handle_connect 0x7f9ee40b4670 2024-03-25T19:17:39.464 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.464+0000 7f9ee17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:17:39.470 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.470+0000 7f9ee17fa640 10 client.?.objecter ms_handle_connect 0x7f9ee413c480 2024-03-25T19:17:39.470 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.470+0000 7f9ee17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:17:39.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.474+0000 7f9ee17fa640 1 -- 172.21.15.6:0/707600607 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 606295301 0 0) 0x7f9ee4130cc0 con 0x7f9ee40b4670 2024-03-25T19:17:39.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.474+0000 7f9ee17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:17:39.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.474+0000 7f9ee17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:39.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.474+0000 7f9ee17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:39.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.474+0000 7f9ee17fa640 10 monclient(hunting): my global_id is 4382 2024-03-25T19:17:39.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.474+0000 7f9ee17fa640 1 -- 172.21.15.6:0/707600607 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ec8004ee0 con 0x7f9ee40b4670 2024-03-25T19:17:39.480 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.481+0000 7f9ee17fa640 1 -- 172.21.15.6:0/707600607 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2636738562 0 0) 0x7f9ee4146570 con 0x7f9ee40b4ff0 2024-03-25T19:17:39.480 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.481+0000 7f9ee17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:17:39.481 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.481+0000 7f9ee17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:39.481 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.481+0000 7f9ee17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:39.481 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.481+0000 7f9ee17fa640 10 monclient(hunting): my global_id is 4384 2024-03-25T19:17:39.481 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.481+0000 7f9ee17fa640 1 -- 172.21.15.6:0/707600607 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ee4130cc0 con 0x7f9ee40b4ff0 2024-03-25T19:17:39.484 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.484+0000 7f9ee17fa640 1 -- 172.21.15.6:0/707600607 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1495039010 0 0) 0x7f9ee4143c80 con 0x7f9ee413c480 2024-03-25T19:17:39.484 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.484+0000 7f9ee17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:17:39.484 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.484+0000 7f9ee17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:39.484 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.484+0000 7f9ee17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:39.484 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.484+0000 7f9ee17fa640 10 monclient(hunting): my global_id is 4380 2024-03-25T19:17:39.484 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.484+0000 7f9ee17fa640 1 -- 172.21.15.6:0/707600607 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ee4146570 con 0x7f9ee413c480 2024-03-25T19:17:39.497 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.497+0000 7f9ee17fa640 1 -- 172.21.15.6:0/707600607 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3712319860 0 0) 0x7f9ec8004ee0 con 0x7f9ee40b4670 2024-03-25T19:17:39.497 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.497+0000 7f9ee17fa640 1 -- 172.21.15.6:0/707600607 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ee4143c80 con 0x7f9ee40b4670 2024-03-25T19:17:39.497 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.497+0000 7f9ee17fa640 1 -- 172.21.15.6:0/707600607 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f9ed8001b00 con 0x7f9ee40b4670 2024-03-25T19:17:39.497 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.497+0000 7f9ee17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:17:39.497 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.497+0000 7f9ee17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:17:39.497 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.497+0000 7f9ee17fa640 10 monclient(hunting): dump: 2024-03-25T19:17:39.497 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:17:39.497 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:39.497 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:39.497 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:39.497 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:17:39.497 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:39.498 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:17:39.498 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:17:39.498 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:17:39.498 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:39.498 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.497+0000 7f9ee17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:17:39.507 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.507+0000 7f9ee17fa640 1 -- 172.21.15.6:0/707600607 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 975433898 0 0) 0x7f9ee4143c80 con 0x7f9ee40b4670 2024-03-25T19:17:39.507 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.507+0000 7f9ee17fa640 1 -- 172.21.15.6:0/707600607 >> v1:172.21.15.192:6789/0 conn(0x7f9ee40b4ff0 legacy=0x7f9ee4145e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:39.507 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.507+0000 7f9ee17fa640 1 -- 172.21.15.6:0/707600607 >> v1:172.21.15.6:6789/0 conn(0x7f9ee413c480 legacy=0x7f9ee41e2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:39.507 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.507+0000 7f9ee17fa640 10 monclient: _finish_hunting 0 2024-03-25T19:17:39.507 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.507+0000 7f9ee17fa640 1 monclient: found mon.c 2024-03-25T19:17:39.507 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.507+0000 7f9ee17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:17:39.507 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.507+0000 7f9ee17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:17:39.508 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.507+0000 7f9ee17fa640 1 -- 172.21.15.6:0/707600607 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee4144a00 con 0x7f9ee40b4670 2024-03-25T19:17:39.508 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.507+0000 7f9ee17fa640 10 monclient: _finish_auth 0 2024-03-25T19:17:39.508 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.507+0000 7f9ee17fa640 10 monclient: _check_auth_tickets 2024-03-25T19:17:39.508 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.507+0000 7f9ee17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:17:39.508 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.507+0000 7f9eeb057640 5 monclient: authenticate success, global_id 4382 2024-03-25T19:17:39.508 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.507+0000 7f9eeb057640 10 monclient: _renew_subs 2024-03-25T19:17:39.508 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.507+0000 7f9eeb057640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:17:39.508 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.507+0000 7f9eeb057640 1 -- 172.21.15.6:0/707600607 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ee4144ce0 con 0x7f9ee40b4670 2024-03-25T19:17:39.508 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.507+0000 7f9eeb057640 10 client.4382.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:17:39.508 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.507+0000 7f9eeb057640 10 monclient: _renew_subs 2024-03-25T19:17:39.508 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.507+0000 7f9eeb057640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:17:39.508 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.507+0000 7f9eeb057640 1 -- 172.21.15.6:0/707600607 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ee413eec0 con 0x7f9ee40b4670 2024-03-25T19:17:39.508 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.507+0000 7f9eeb057640 1 librados: init done 2024-03-25T19:17:39.509 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.508+0000 7f9eeb057640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:17:39.509 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.508+0000 7f9eeb057640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:17:39.509 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.508+0000 7f9eeb057640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:17:39.509 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.508+0000 7f9eeb057640 1 -- 172.21.15.6:0/707600607 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ee406dbd0 con 0x7f9ee40b4670 2024-03-25T19:17:39.824 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.824+0000 7f9ee17fa640 1 -- 172.21.15.6:0/707600607 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ed8004a90 con 0x7f9ee40b4670 2024-03-25T19:17:39.825 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.824+0000 7f9ee17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:17:39.825 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.824+0000 7f9ee17fa640 1 -- 172.21.15.6:0/707600607 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f9ed8004da0 con 0x7f9ee40b4670 2024-03-25T19:17:39.825 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.824+0000 7f9ee17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:17:39.825 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.824+0000 7f9ee17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:17:39.825 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.824+0000 7f9ee17fa640 10 monclient: dump: 2024-03-25T19:17:39.825 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:17:39.825 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:39.825 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:39.825 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:39.825 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:17:39.825 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:39.825 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:17:39.825 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:17:39.825 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:17:39.825 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:39.826 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.824+0000 7f9ee2ffd640 4 set_mon_vals no callback set 2024-03-25T19:17:39.826 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.825+0000 7f9ee17fa640 1 -- 172.21.15.6:0/707600607 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f9ed801ad70 con 0x7f9ee40b4670 2024-03-25T19:17:39.826 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.826+0000 7f9ee3fff640 10 monclient: get_auth_request con 0x7f9ec806de50 auth_method 0 2024-03-25T19:17:39.826 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.826+0000 7f9ee17fa640 10 client.4382.objecter ms_handle_connect 0x7f9ec806de50 2024-03-25T19:17:39.831 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.831+0000 7f9ee17fa640 1 -- 172.21.15.6:0/707600607 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f9ed8054870 con 0x7f9ee40b4670 2024-03-25T19:17:39.831 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.831+0000 7f9ee17fa640 10 client.4382.objecter ms_dispatch 0x7f9ee4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:17:39.831 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.831+0000 7f9ee17fa640 3 client.4382.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:17:39.831 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.831+0000 7f9ee17fa640 3 client.4382.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:17:39.831 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.831+0000 7f9ee17fa640 20 client.4382.objecter dump_active .. 0 homeless 2024-03-25T19:17:39.831 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.831+0000 7f9ee17fa640 1 -- 172.21.15.6:0/707600607 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9ed8050380 con 0x7f9ee40b4670 2024-03-25T19:17:39.831 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.831+0000 7f9ee17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:17:39.831 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:39.831+0000 7f9ee17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:17:40.070 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:40.070+0000 7f9eeb057640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:17:40.070 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:40.070+0000 7f9eeb057640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:17:40.071 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:40.070+0000 7f9eeb057640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:17:40.071 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:40.070+0000 7f9eeb057640 1 -- 172.21.15.6:0/707600607 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ee40688c0 con 0x7f9ee40b4670 2024-03-25T19:17:40.125 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:40.125+0000 7f9ee17fa640 1 -- 172.21.15.6:0/707600607 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f9ed80522b0 con 0x7f9ee40b4670 2024-03-25T19:17:40.126 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:40.125+0000 7f9ee17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:17:40.126 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:40.125+0000 7f9ee17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:17:40.126 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:17:40.126 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:17:40.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:40.128+0000 7f9eeb057640 10 librados: watch_flush enter 2024-03-25T19:17:40.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:40.128+0000 7f9eeb057640 10 librados: watch_flush exit 2024-03-25T19:17:40.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:40.128+0000 7f9eeb057640 20 client.4382.objecter shutdown clearing up homeless session... 2024-03-25T19:17:40.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:40.128+0000 7f9eeb057640 10 client.4382.objecter successfully canceled tick 2024-03-25T19:17:40.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:40.128+0000 7f9eeb057640 1 -- 172.21.15.6:0/707600607 >> v1:172.21.15.6:6800/2936652650 conn(0x7f9ec806de50 legacy=0x7f9ec8070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:40.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:40.128+0000 7f9eeb057640 10 monclient: shutdown 2024-03-25T19:17:40.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:40.128+0000 7f9eeb057640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:17:40.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:40.128+0000 7f9eeb057640 1 -- 172.21.15.6:0/707600607 >> v1:172.21.15.6:6790/0 conn(0x7f9ee40b4670 legacy=0x7f9ee4145750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:40.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:40.128+0000 7f9eeb057640 1 -- 172.21.15.6:0/707600607 shutdown_connections 2024-03-25T19:17:40.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:40.128+0000 7f9eeb057640 1 -- 172.21.15.6:0/707600607 >> 172.21.15.6:0/707600607 conn(0x7f9ee40617a0 msgr2=0x7f9ee40b2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:17:40.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:40.128+0000 7f9eeb057640 1 -- 172.21.15.6:0/707600607 shutdown_connections 2024-03-25T19:17:40.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:40.128+0000 7f9eeb057640 1 -- 172.21.15.6:0/707600607 wait complete. 2024-03-25T19:17:40.130 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:40.129+0000 7f9eeb057640 1 librados: shutdown 2024-03-25T19:17:40.139 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:17:42.810 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:04:00.157 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:17:42.817 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:04:00.164 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:17:42.851 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:04:00.198 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:17:42.881 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:04:00.228 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:17:42.887 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:04:00.233 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:17:42.894 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:04:00.212 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:17:42.904 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:04:00.222 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:17:42.929 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:04:00.276 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:17:42.934 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:04:00.252 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:17:46.945 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:17:47.044 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.044+0000 7f894dfd0640 10 monclient: get_monmap_and_config 2024-03-25T19:17:47.044 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.044+0000 7f894dfd0640 10 monclient: build_initial_monmap 2024-03-25T19:17:47.044 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.044+0000 7f894dfd0640 10 monclient: monmap: 2024-03-25T19:17:47.044 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:17:47.044 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:47.044 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:17:47.045155+0000 2024-03-25T19:17:47.044 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:17:47.045155+0000 2024-03-25T19:17:47.044 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:17:47.044 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:47.044 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:17:47.044 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:17:47.044 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:17:47.045 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:47.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.045+0000 7f894dfd0640 1 Processor -- start 2024-03-25T19:17:47.046 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.045+0000 7f894dfd0640 1 -- start start 2024-03-25T19:17:47.046 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.045+0000 7f894dfd0640 10 monclient: init 2024-03-25T19:17:47.046 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.045+0000 7f894dfd0640 10 monclient: _reopen_session rank -1 2024-03-25T19:17:47.046 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.045+0000 7f894dfd0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:17:47.046 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.045+0000 7f894dfd0640 10 monclient(hunting): picked mon.noname-c con 0x7f89480b4ff0 addr v1:172.21.15.192:6789/0 2024-03-25T19:17:47.046 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.045+0000 7f894dfd0640 10 monclient(hunting): picked mon.noname-b con 0x7f89480b4670 addr v1:172.21.15.6:6790/0 2024-03-25T19:17:47.046 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.045+0000 7f894dfd0640 10 monclient(hunting): picked mon.noname-a con 0x7f894813c480 addr v1:172.21.15.6:6789/0 2024-03-25T19:17:47.046 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.045+0000 7f894dfd0640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89480ab0e0 con 0x7f894813c480 2024-03-25T19:17:47.046 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.045+0000 7f894dfd0640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89480ab3c0 con 0x7f89480b4ff0 2024-03-25T19:17:47.046 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.045+0000 7f894dfd0640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89480a9f30 con 0x7f89480b4670 2024-03-25T19:17:47.046 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.045+0000 7f894dfd0640 10 monclient(hunting): _renew_subs 2024-03-25T19:17:47.046 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.045+0000 7f894dfd0640 10 monclient(hunting): authenticate will time out at 6171.285511s 2024-03-25T19:17:47.053 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.053+0000 7f89477fe640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f89480b4ff0 0x7f894813bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:54882/0 (socket says 172.21.15.6:54882) 2024-03-25T19:17:47.053 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.053+0000 7f89477fe640 1 -- 172.21.15.6:0/1723539399 learned_addr learned my addr 172.21.15.6:0/1723539399 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:17:47.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.065+0000 7f89467fc640 1 -- 172.21.15.6:0/1723539399 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4230606287 0 0) 0x7f89480ab3c0 con 0x7f89480b4ff0 2024-03-25T19:17:47.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.065+0000 7f89467fc640 10 monclient(hunting): _negotiate 2024-03-25T19:17:47.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.065+0000 7f89467fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:47.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.065+0000 7f89467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:47.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.065+0000 7f89467fc640 10 monclient(hunting): my global_id is 4387 2024-03-25T19:17:47.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.065+0000 7f89467fc640 1 -- 172.21.15.6:0/1723539399 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8930003520 con 0x7f89480b4ff0 2024-03-25T19:17:47.066 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.066+0000 7f89467fc640 1 -- 172.21.15.6:0/1723539399 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4153320026 0 0) 0x7f89480ab0e0 con 0x7f894813c480 2024-03-25T19:17:47.066 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.066+0000 7f89467fc640 10 monclient(hunting): _negotiate 2024-03-25T19:17:47.066 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.066+0000 7f89467fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:47.066 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.066+0000 7f89467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:47.066 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.066+0000 7f89467fc640 10 monclient(hunting): my global_id is 4383 2024-03-25T19:17:47.066 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.066+0000 7f89467fc640 1 -- 172.21.15.6:0/1723539399 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89480ab3c0 con 0x7f894813c480 2024-03-25T19:17:47.077 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.077+0000 7f89467fc640 1 -- 172.21.15.6:0/1723539399 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2474733322 0 0) 0x7f89480a9f30 con 0x7f89480b4670 2024-03-25T19:17:47.077 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.077+0000 7f89467fc640 10 monclient(hunting): _negotiate 2024-03-25T19:17:47.077 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.077+0000 7f89467fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:47.077 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.077+0000 7f89467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:47.077 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.077+0000 7f89467fc640 10 monclient(hunting): my global_id is 4385 2024-03-25T19:17:47.077 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.077+0000 7f89467fc640 1 -- 172.21.15.6:0/1723539399 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89480ab0e0 con 0x7f89480b4670 2024-03-25T19:17:47.093 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.092+0000 7f89467fc640 1 -- 172.21.15.6:0/1723539399 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2246994665 0 0) 0x7f8930003520 con 0x7f89480b4ff0 2024-03-25T19:17:47.093 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.092+0000 7f89467fc640 1 -- 172.21.15.6:0/1723539399 >> v1:172.21.15.6:6790/0 conn(0x7f89480b4670 legacy=0x7f89481425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:47.093 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.092+0000 7f89467fc640 1 -- 172.21.15.6:0/1723539399 >> v1:172.21.15.6:6789/0 conn(0x7f894813c480 legacy=0x7f894814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:47.093 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.092+0000 7f89467fc640 10 monclient: _finish_hunting 0 2024-03-25T19:17:47.093 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.092+0000 7f89467fc640 1 monclient: found mon.noname-c 2024-03-25T19:17:47.093 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.092+0000 7f89467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:17:47.093 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.092+0000 7f89467fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:17:47.093 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.092+0000 7f89467fc640 1 -- 172.21.15.6:0/1723539399 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89480a9b70 con 0x7f89480b4ff0 2024-03-25T19:17:47.093 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.092+0000 7f89467fc640 10 monclient: _finish_auth 0 2024-03-25T19:17:47.093 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.092+0000 7f89467fc640 10 monclient: _check_auth_tickets 2024-03-25T19:17:47.093 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.092+0000 7f89467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:17:47.093 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.092+0000 7f89467fc640 1 -- 172.21.15.6:0/1723539399 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f8938001690 con 0x7f89480b4ff0 2024-03-25T19:17:47.093 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.092+0000 7f89467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:17:47.094 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.092+0000 7f89467fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:17:47.094 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.092+0000 7f89467fc640 10 monclient: dump: 2024-03-25T19:17:47.094 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:17:47.094 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:47.094 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:47.094 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:47.094 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:17:47.094 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:47.094 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:17:47.094 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:17:47.094 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:17:47.094 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:47.094 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.092+0000 7f894dfd0640 5 monclient: authenticate success, global_id 4387 2024-03-25T19:17:47.094 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.092+0000 7f894dfd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:17:47.123 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.123+0000 7f89467fc640 1 -- 172.21.15.6:0/1723539399 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89380017f0 con 0x7f89480b4ff0 2024-03-25T19:17:47.123 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.123+0000 7f89467fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:17:47.123 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.123+0000 7f89467fc640 1 -- 172.21.15.6:0/1723539399 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f8938001c00 con 0x7f89480b4ff0 2024-03-25T19:17:47.123 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.123+0000 7f894dfd0640 10 monclient: get_monmap_and_config success 2024-03-25T19:17:47.123 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.123+0000 7f894dfd0640 4 set_mon_vals no callback set 2024-03-25T19:17:47.123 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.123+0000 7f89467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:17:47.123 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.123+0000 7f89467fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:17:47.123 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.123+0000 7f89467fc640 10 monclient: dump: 2024-03-25T19:17:47.123 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:17:47.124 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:47.124 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:47.124 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:47.124 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:17:47.124 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:47.124 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:17:47.124 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:17:47.124 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:17:47.124 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:47.125 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.124+0000 7f894dfd0640 10 monclient: shutdown 2024-03-25T19:17:47.125 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.124+0000 7f894dfd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:17:47.125 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.124+0000 7f894dfd0640 1 -- 172.21.15.6:0/1723539399 >> v1:172.21.15.192:6789/0 conn(0x7f89480b4ff0 legacy=0x7f894813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:47.125 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.124+0000 7f894dfd0640 1 -- 172.21.15.6:0/1723539399 shutdown_connections 2024-03-25T19:17:47.126 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.124+0000 7f894dfd0640 1 -- 172.21.15.6:0/1723539399 >> 172.21.15.6:0/1723539399 conn(0x7f89480617a0 msgr2=0x7f8948061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:17:47.126 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.124+0000 7f894dfd0640 1 -- 172.21.15.6:0/1723539399 shutdown_connections 2024-03-25T19:17:47.126 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.124+0000 7f894dfd0640 1 -- 172.21.15.6:0/1723539399 wait complete. 2024-03-25T19:17:47.126 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.125+0000 7f894dfd0640 10 monclient: build_initial_monmap 2024-03-25T19:17:47.126 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.125+0000 7f894dfd0640 10 monclient: monmap: 2024-03-25T19:17:47.126 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:17:47.126 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:17:47.126 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:17:47.126 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:17:47.126 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:17:47.126 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:17:47.126 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:17:47.126 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:17:47.127 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:17:47.127 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:47.127 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.125+0000 7f894dfd0640 1 librados: starting msgr at 2024-03-25T19:17:47.127 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.125+0000 7f894dfd0640 1 librados: starting objecter 2024-03-25T19:17:47.127 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.126+0000 7f894dfd0640 1 Processor -- start 2024-03-25T19:17:47.128 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.126+0000 7f894dfd0640 1 -- start start 2024-03-25T19:17:47.128 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.126+0000 7f894dfd0640 1 librados: setting wanted keys 2024-03-25T19:17:47.128 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.126+0000 7f894dfd0640 1 librados: calling monclient init 2024-03-25T19:17:47.128 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.126+0000 7f894dfd0640 10 monclient: init 2024-03-25T19:17:47.128 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.127+0000 7f894dfd0640 10 monclient: _reopen_session rank -1 2024-03-25T19:17:47.128 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.127+0000 7f894dfd0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:17:47.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.127+0000 7f894dfd0640 10 monclient(hunting): picked mon.noname-c con 0x7f89480b4670 addr v1:172.21.15.6:6790/0 2024-03-25T19:17:47.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.127+0000 7f894dfd0640 10 monclient(hunting): picked mon.noname-b con 0x7f89480b4ff0 addr v1:172.21.15.192:6789/0 2024-03-25T19:17:47.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.127+0000 7f894dfd0640 10 monclient(hunting): picked mon.noname-a con 0x7f894813c480 addr v1:172.21.15.6:6789/0 2024-03-25T19:17:47.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.127+0000 7f894dfd0640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89480abc10 con 0x7f894813c480 2024-03-25T19:17:47.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.127+0000 7f894dfd0640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89480ac380 con 0x7f89480b4ff0 2024-03-25T19:17:47.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.127+0000 7f894dfd0640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89480ac660 con 0x7f89480b4670 2024-03-25T19:17:47.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.127+0000 7f894dfd0640 10 monclient(hunting): _renew_subs 2024-03-25T19:17:47.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.127+0000 7f894dfd0640 10 monclient(hunting): authenticate will time out at 6171.367123s 2024-03-25T19:17:47.136 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.136+0000 7f8946ffd640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f89480b4ff0 0x7f89481d81e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:54938/0 (socket says 172.21.15.6:54938) 2024-03-25T19:17:47.136 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.136+0000 7f8946ffd640 1 -- 172.21.15.6:0/3501285187 learned_addr learned my addr 172.21.15.6:0/3501285187 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:17:47.137 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.137+0000 7f892ffff640 10 client.?.objecter ms_handle_connect 0x7f89480b4ff0 2024-03-25T19:17:47.137 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.137+0000 7f892ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:17:47.143 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.143+0000 7f892ffff640 1 -- 172.21.15.6:0/3501285187 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 279457886 0 0) 0x7f89480ac380 con 0x7f89480b4ff0 2024-03-25T19:17:47.143 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.143+0000 7f892ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:17:47.143 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.143+0000 7f892ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:47.143 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.143+0000 7f892ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:47.144 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.143+0000 7f892ffff640 10 monclient(hunting): my global_id is 4390 2024-03-25T19:17:47.144 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.143+0000 7f892ffff640 1 -- 172.21.15.6:0/3501285187 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8934004eb0 con 0x7f89480b4ff0 2024-03-25T19:17:47.150 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.150+0000 7f892ffff640 10 client.?.objecter ms_handle_connect 0x7f89480b4670 2024-03-25T19:17:47.150 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.150+0000 7f892ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:17:47.152 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.152+0000 7f892ffff640 1 -- 172.21.15.6:0/3501285187 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1436051426 0 0) 0x7f8934004eb0 con 0x7f89480b4ff0 2024-03-25T19:17:47.152 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.152+0000 7f892ffff640 1 -- 172.21.15.6:0/3501285187 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89480ac380 con 0x7f89480b4ff0 2024-03-25T19:17:47.153 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.152+0000 7f892ffff640 1 -- 172.21.15.6:0/3501285187 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f893c001a10 con 0x7f89480b4ff0 2024-03-25T19:17:47.153 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.152+0000 7f892ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:17:47.153 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.152+0000 7f892ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:17:47.153 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.152+0000 7f892ffff640 10 monclient(hunting): dump: 2024-03-25T19:17:47.153 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:17:47.153 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:47.153 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:47.153 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:47.153 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:17:47.153 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:47.153 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:17:47.153 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:17:47.153 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:17:47.153 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:47.154 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.152+0000 7f892ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:17:47.157 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.157+0000 7f892ffff640 10 client.?.objecter ms_handle_connect 0x7f894813c480 2024-03-25T19:17:47.157 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.157+0000 7f892ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:17:47.166 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.165+0000 7f892ffff640 1 -- 172.21.15.6:0/3501285187 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3567016014 0 0) 0x7f89480ac380 con 0x7f89480b4ff0 2024-03-25T19:17:47.166 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.165+0000 7f892ffff640 1 -- 172.21.15.6:0/3501285187 >> v1:172.21.15.6:6790/0 conn(0x7f89480b4670 legacy=0x7f8948137070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:47.166 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.165+0000 7f892ffff640 1 -- 172.21.15.6:0/3501285187 >> v1:172.21.15.6:6789/0 conn(0x7f894813c480 legacy=0x7f89481e3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:47.166 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.165+0000 7f892ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:17:47.166 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.165+0000 7f892ffff640 1 monclient: found mon.b 2024-03-25T19:17:47.166 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.165+0000 7f892ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:17:47.166 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.165+0000 7f892ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:17:47.166 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.165+0000 7f892ffff640 1 -- 172.21.15.6:0/3501285187 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89480ac940 con 0x7f89480b4ff0 2024-03-25T19:17:47.166 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.165+0000 7f892ffff640 10 monclient: _finish_auth 0 2024-03-25T19:17:47.166 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.165+0000 7f892ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:17:47.167 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.165+0000 7f892ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:17:47.167 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.165+0000 7f894dfd0640 5 monclient: authenticate success, global_id 4390 2024-03-25T19:17:47.167 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.165+0000 7f894dfd0640 10 monclient: _renew_subs 2024-03-25T19:17:47.167 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.165+0000 7f894dfd0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:17:47.167 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.165+0000 7f894dfd0640 1 -- 172.21.15.6:0/3501285187 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89480ab8e0 con 0x7f89480b4ff0 2024-03-25T19:17:47.167 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.165+0000 7f894dfd0640 10 client.4390.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:17:47.167 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.165+0000 7f894dfd0640 10 monclient: _renew_subs 2024-03-25T19:17:47.167 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.165+0000 7f894dfd0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:17:47.167 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.165+0000 7f894dfd0640 1 -- 172.21.15.6:0/3501285187 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f89480ac130 con 0x7f89480b4ff0 2024-03-25T19:17:47.167 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.165+0000 7f894dfd0640 1 librados: init done 2024-03-25T19:17:47.167 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.167+0000 7f894dfd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:17:47.168 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.167+0000 7f894dfd0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:17:47.168 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.167+0000 7f894dfd0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:17:47.168 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.167+0000 7f894dfd0640 1 -- 172.21.15.6:0/3501285187 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89480b3ff0 con 0x7f89480b4ff0 2024-03-25T19:17:47.439 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.439+0000 7f892ffff640 1 -- 172.21.15.6:0/3501285187 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f893c001cf0 con 0x7f89480b4ff0 2024-03-25T19:17:47.439 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.439+0000 7f892ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:17:47.439 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.439+0000 7f892ffff640 1 -- 172.21.15.6:0/3501285187 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f893c003a60 con 0x7f89480b4ff0 2024-03-25T19:17:47.439 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.439+0000 7f892ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:17:47.439 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.439+0000 7f892ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:17:47.439 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.439+0000 7f892ffff640 10 monclient: dump: 2024-03-25T19:17:47.439 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:17:47.440 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:47.440 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:47.440 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:47.440 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:17:47.440 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:47.440 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:17:47.440 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:17:47.440 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:17:47.440 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:47.440 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.439+0000 7f8945ffb640 4 set_mon_vals no callback set 2024-03-25T19:17:47.441 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.440+0000 7f892ffff640 1 -- 172.21.15.6:0/3501285187 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f893c003c80 con 0x7f89480b4ff0 2024-03-25T19:17:47.441 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.441+0000 7f89477fe640 10 monclient: get_auth_request con 0x7f893406dc70 auth_method 0 2024-03-25T19:17:47.441 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.441+0000 7f892ffff640 10 client.4390.objecter ms_handle_connect 0x7f893406dc70 2024-03-25T19:17:47.445 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.445+0000 7f892ffff640 1 -- 172.21.15.6:0/3501285187 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f893c0548c0 con 0x7f89480b4ff0 2024-03-25T19:17:47.445 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.445+0000 7f892ffff640 10 client.4390.objecter ms_dispatch 0x7f8948000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:17:47.445 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.445+0000 7f892ffff640 3 client.4390.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:17:47.445 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.445+0000 7f892ffff640 3 client.4390.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:17:47.445 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.445+0000 7f892ffff640 20 client.4390.objecter dump_active .. 0 homeless 2024-03-25T19:17:47.446 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.445+0000 7f892ffff640 1 -- 172.21.15.6:0/3501285187 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f893c0503d0 con 0x7f89480b4ff0 2024-03-25T19:17:47.446 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.445+0000 7f892ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:17:47.446 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.445+0000 7f892ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:17:47.697 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.697+0000 7f894dfd0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:17:47.698 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.697+0000 7f894dfd0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:17:47.698 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.697+0000 7f894dfd0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:17:47.698 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.697+0000 7f894dfd0640 1 -- 172.21.15.6:0/3501285187 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89480aba70 con 0x7f89480b4ff0 2024-03-25T19:17:47.724 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.724+0000 7f892ffff640 1 -- 172.21.15.6:0/3501285187 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f89480aba70 con 0x7f89480b4ff0 2024-03-25T19:17:47.724 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.724+0000 7f892ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:17:47.724 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.724+0000 7f892ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:17:47.725 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:17:47.725 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:17:47.727 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.727+0000 7f894dfd0640 10 librados: watch_flush enter 2024-03-25T19:17:47.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.727+0000 7f894dfd0640 10 librados: watch_flush exit 2024-03-25T19:17:47.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.727+0000 7f894dfd0640 20 client.4390.objecter shutdown clearing up homeless session... 2024-03-25T19:17:47.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.727+0000 7f894dfd0640 10 client.4390.objecter successfully canceled tick 2024-03-25T19:17:47.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.727+0000 7f894dfd0640 1 -- 172.21.15.6:0/3501285187 >> v1:172.21.15.6:6800/2936652650 conn(0x7f893406dc70 legacy=0x7f8934070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:47.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.727+0000 7f894dfd0640 10 monclient: shutdown 2024-03-25T19:17:47.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.727+0000 7f894dfd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:17:47.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.727+0000 7f894dfd0640 1 -- 172.21.15.6:0/3501285187 >> v1:172.21.15.192:6789/0 conn(0x7f89480b4ff0 legacy=0x7f89481d81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:47.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.727+0000 7f894dfd0640 1 -- 172.21.15.6:0/3501285187 shutdown_connections 2024-03-25T19:17:47.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.727+0000 7f894dfd0640 1 -- 172.21.15.6:0/3501285187 >> 172.21.15.6:0/3501285187 conn(0x7f89480617a0 msgr2=0x7f8948145420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:17:47.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.727+0000 7f894dfd0640 1 -- 172.21.15.6:0/3501285187 shutdown_connections 2024-03-25T19:17:47.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.727+0000 7f894dfd0640 1 -- 172.21.15.6:0/3501285187 wait complete. 2024-03-25T19:17:47.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:47.727+0000 7f894dfd0640 1 librados: shutdown 2024-03-25T19:17:47.738 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:17:54.543 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:17:54.664 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.664+0000 7f239bc84640 10 monclient: get_monmap_and_config 2024-03-25T19:17:54.664 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.664+0000 7f239bc84640 10 monclient: build_initial_monmap 2024-03-25T19:17:54.664 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.664+0000 7f239bc84640 10 monclient: monmap: 2024-03-25T19:17:54.664 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:17:54.664 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:54.664 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:17:54.665283+0000 2024-03-25T19:17:54.664 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:17:54.665283+0000 2024-03-25T19:17:54.664 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:17:54.664 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:54.664 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:17:54.664 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:17:54.665 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:17:54.665 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:54.666 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.664+0000 7f239bc84640 1 Processor -- start 2024-03-25T19:17:54.666 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.665+0000 7f239bc84640 1 -- start start 2024-03-25T19:17:54.666 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.665+0000 7f239bc84640 10 monclient: init 2024-03-25T19:17:54.666 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.665+0000 7f239bc84640 10 monclient: _reopen_session rank -1 2024-03-25T19:17:54.666 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.665+0000 7f239bc84640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:17:54.666 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.665+0000 7f239bc84640 10 monclient(hunting): picked mon.noname-c con 0x7f23940b4ff0 addr v1:172.21.15.192:6789/0 2024-03-25T19:17:54.666 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.665+0000 7f239bc84640 10 monclient(hunting): picked mon.noname-a con 0x7f23940b4670 addr v1:172.21.15.6:6789/0 2024-03-25T19:17:54.666 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.665+0000 7f239bc84640 10 monclient(hunting): picked mon.noname-b con 0x7f239413c480 addr v1:172.21.15.6:6790/0 2024-03-25T19:17:54.666 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.665+0000 7f239bc84640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23940ab0e0 con 0x7f23940b4670 2024-03-25T19:17:54.666 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.665+0000 7f239bc84640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23940ab3c0 con 0x7f23940b4ff0 2024-03-25T19:17:54.666 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.665+0000 7f239bc84640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23940a9f30 con 0x7f239413c480 2024-03-25T19:17:54.666 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.665+0000 7f239bc84640 10 monclient(hunting): _renew_subs 2024-03-25T19:17:54.667 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.665+0000 7f239bc84640 10 monclient(hunting): authenticate will time out at 6178.905572s 2024-03-25T19:17:54.680 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.680+0000 7f239921f640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f23940b4670 0x7f23941425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43690/0 (socket says 172.21.15.6:43690) 2024-03-25T19:17:54.680 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.680+0000 7f239921f640 1 -- 172.21.15.6:0/741961321 learned_addr learned my addr 172.21.15.6:0/741961321 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:17:54.690 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.690+0000 7f2398a1e640 1 -- 172.21.15.6:0/741961321 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3515171853 0 0) 0x7f23940ab3c0 con 0x7f23940b4ff0 2024-03-25T19:17:54.690 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.690+0000 7f2398a1e640 10 monclient(hunting): _negotiate 2024-03-25T19:17:54.690 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.690+0000 7f2398a1e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:54.690 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.690+0000 7f2398a1e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:54.690 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.690+0000 7f2398a1e640 10 monclient(hunting): my global_id is 4393 2024-03-25T19:17:54.690 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.690+0000 7f2398a1e640 1 -- 172.21.15.6:0/741961321 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2380003520 con 0x7f23940b4ff0 2024-03-25T19:17:54.695 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.694+0000 7f2398a1e640 1 -- 172.21.15.6:0/741961321 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3692629106 0 0) 0x7f23940a9f30 con 0x7f239413c480 2024-03-25T19:17:54.695 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.694+0000 7f2398a1e640 10 monclient(hunting): _negotiate 2024-03-25T19:17:54.695 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.694+0000 7f2398a1e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:54.695 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.694+0000 7f2398a1e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:54.695 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.694+0000 7f2398a1e640 10 monclient(hunting): my global_id is 4391 2024-03-25T19:17:54.695 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.694+0000 7f2398a1e640 1 -- 172.21.15.6:0/741961321 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23940ab3c0 con 0x7f239413c480 2024-03-25T19:17:54.695 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.695+0000 7f2398a1e640 1 -- 172.21.15.6:0/741961321 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2601276066 0 0) 0x7f23940ab0e0 con 0x7f23940b4670 2024-03-25T19:17:54.695 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.695+0000 7f2398a1e640 10 monclient(hunting): _negotiate 2024-03-25T19:17:54.695 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.695+0000 7f2398a1e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:54.695 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.695+0000 7f2398a1e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:54.695 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.695+0000 7f2398a1e640 10 monclient(hunting): my global_id is 4389 2024-03-25T19:17:54.695 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.695+0000 7f2398a1e640 1 -- 172.21.15.6:0/741961321 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23940a9f30 con 0x7f23940b4670 2024-03-25T19:17:54.706 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.706+0000 7f2398a1e640 1 -- 172.21.15.6:0/741961321 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 380258244 0 0) 0x7f23940ab3c0 con 0x7f239413c480 2024-03-25T19:17:54.706 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.706+0000 7f2398a1e640 1 -- 172.21.15.6:0/741961321 >> v1:172.21.15.192:6789/0 conn(0x7f23940b4ff0 legacy=0x7f239413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:54.706 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.706+0000 7f2398a1e640 1 -- 172.21.15.6:0/741961321 >> v1:172.21.15.6:6789/0 conn(0x7f23940b4670 legacy=0x7f23941425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:54.707 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.706+0000 7f2398a1e640 10 monclient: _finish_hunting 0 2024-03-25T19:17:54.707 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.706+0000 7f2398a1e640 1 monclient: found mon.noname-b 2024-03-25T19:17:54.707 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.706+0000 7f2398a1e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:17:54.707 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.706+0000 7f2398a1e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-03-25T19:17:54.707 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.706+0000 7f2398a1e640 1 -- 172.21.15.6:0/741961321 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23940a9b70 con 0x7f239413c480 2024-03-25T19:17:54.707 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.706+0000 7f2398a1e640 10 monclient: _finish_auth 0 2024-03-25T19:17:54.707 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.706+0000 7f2398a1e640 10 monclient: _check_auth_tickets 2024-03-25T19:17:54.707 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.706+0000 7f2398a1e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:17:54.707 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.706+0000 7f2398a1e640 1 -- 172.21.15.6:0/741961321 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f238c0019f0 con 0x7f239413c480 2024-03-25T19:17:54.707 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.706+0000 7f2398a1e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:17:54.707 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.706+0000 7f2398a1e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:17:54.707 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.706+0000 7f2398a1e640 10 monclient: dump: 2024-03-25T19:17:54.707 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:17:54.707 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:54.707 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:54.708 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:54.708 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:17:54.708 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:54.708 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:17:54.708 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:17:54.708 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:17:54.708 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:54.708 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.706+0000 7f239bc84640 5 monclient: authenticate success, global_id 4391 2024-03-25T19:17:54.708 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.706+0000 7f239bc84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:17:54.720 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.720+0000 7f2398a1e640 1 -- 172.21.15.6:0/741961321 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f238c001b50 con 0x7f239413c480 2024-03-25T19:17:54.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.720+0000 7f2398a1e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:17:54.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.720+0000 7f2398a1e640 1 -- 172.21.15.6:0/741961321 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f238c001f60 con 0x7f239413c480 2024-03-25T19:17:54.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.720+0000 7f239bc84640 10 monclient: get_monmap_and_config success 2024-03-25T19:17:54.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.720+0000 7f239bc84640 4 set_mon_vals no callback set 2024-03-25T19:17:54.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.720+0000 7f2398a1e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:17:54.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.720+0000 7f2398a1e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:17:54.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.720+0000 7f2398a1e640 10 monclient: dump: 2024-03-25T19:17:54.721 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:17:54.721 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:54.721 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:54.721 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:54.721 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:17:54.721 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:54.721 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:17:54.721 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:17:54.722 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:17:54.722 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:54.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.721+0000 7f239bc84640 10 monclient: shutdown 2024-03-25T19:17:54.724 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.721+0000 7f239bc84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:17:54.724 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.721+0000 7f239bc84640 1 -- 172.21.15.6:0/741961321 >> v1:172.21.15.6:6790/0 conn(0x7f239413c480 legacy=0x7f239414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:54.724 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.721+0000 7f239bc84640 1 -- 172.21.15.6:0/741961321 shutdown_connections 2024-03-25T19:17:54.724 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.721+0000 7f239bc84640 1 -- 172.21.15.6:0/741961321 >> 172.21.15.6:0/741961321 conn(0x7f23940617a0 msgr2=0x7f2394061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:17:54.724 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.721+0000 7f239bc84640 1 -- 172.21.15.6:0/741961321 shutdown_connections 2024-03-25T19:17:54.724 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.722+0000 7f239bc84640 1 -- 172.21.15.6:0/741961321 wait complete. 2024-03-25T19:17:54.724 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.722+0000 7f239bc84640 10 monclient: build_initial_monmap 2024-03-25T19:17:54.724 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.722+0000 7f239bc84640 10 monclient: monmap: 2024-03-25T19:17:54.724 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:17:54.724 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:17:54.724 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:17:54.724 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:17:54.724 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:17:54.724 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:17:54.724 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:17:54.724 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:17:54.725 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:17:54.725 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:54.725 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.722+0000 7f239bc84640 1 librados: starting msgr at 2024-03-25T19:17:54.725 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.722+0000 7f239bc84640 1 librados: starting objecter 2024-03-25T19:17:54.725 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.723+0000 7f239bc84640 1 Processor -- start 2024-03-25T19:17:54.725 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.723+0000 7f239bc84640 1 -- start start 2024-03-25T19:17:54.725 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.723+0000 7f239bc84640 1 librados: setting wanted keys 2024-03-25T19:17:54.725 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.723+0000 7f239bc84640 1 librados: calling monclient init 2024-03-25T19:17:54.725 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.723+0000 7f239bc84640 10 monclient: init 2024-03-25T19:17:54.725 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.723+0000 7f239bc84640 10 monclient: _reopen_session rank -1 2024-03-25T19:17:54.725 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.723+0000 7f239bc84640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:17:54.725 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.723+0000 7f239bc84640 10 monclient(hunting): picked mon.noname-b con 0x7f23940b4670 addr v1:172.21.15.192:6789/0 2024-03-25T19:17:54.725 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.723+0000 7f239bc84640 10 monclient(hunting): picked mon.noname-c con 0x7f23940b4ff0 addr v1:172.21.15.6:6790/0 2024-03-25T19:17:54.725 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.723+0000 7f239bc84640 10 monclient(hunting): picked mon.noname-a con 0x7f239413c480 addr v1:172.21.15.6:6789/0 2024-03-25T19:17:54.725 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.723+0000 7f239bc84640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2394143d20 con 0x7f239413c480 2024-03-25T19:17:54.726 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.723+0000 7f239bc84640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2394144a80 con 0x7f23940b4670 2024-03-25T19:17:54.726 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.723+0000 7f239bc84640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2394144d60 con 0x7f23940b4ff0 2024-03-25T19:17:54.726 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.723+0000 7f239bc84640 10 monclient(hunting): _renew_subs 2024-03-25T19:17:54.726 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.723+0000 7f239bc84640 10 monclient(hunting): authenticate will time out at 6178.963437s 2024-03-25T19:17:54.729 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.729+0000 7f2399a20640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f23940b4670 0x7f23941457d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:49360/0 (socket says 172.21.15.6:49360) 2024-03-25T19:17:54.729 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.729+0000 7f2399a20640 1 -- 172.21.15.6:0/2694774487 learned_addr learned my addr 172.21.15.6:0/2694774487 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:17:54.734 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.734+0000 7f238a7fc640 10 client.?.objecter ms_handle_connect 0x7f23940b4670 2024-03-25T19:17:54.734 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.734+0000 7f238a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:17:54.735 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.736+0000 7f238a7fc640 10 client.?.objecter ms_handle_connect 0x7f23940b4ff0 2024-03-25T19:17:54.735 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.736+0000 7f238a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:17:54.742 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.742+0000 7f238a7fc640 1 -- 172.21.15.6:0/2694774487 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1834775755 0 0) 0x7f2394144a80 con 0x7f23940b4670 2024-03-25T19:17:54.742 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.742+0000 7f238a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:17:54.742 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.742+0000 7f238a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:54.742 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.742+0000 7f238a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:54.742 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.742+0000 7f238a7fc640 10 monclient(hunting): my global_id is 4396 2024-03-25T19:17:54.742 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.742+0000 7f238a7fc640 1 -- 172.21.15.6:0/2694774487 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2378004eb0 con 0x7f23940b4670 2024-03-25T19:17:54.742 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.743+0000 7f238a7fc640 10 client.?.objecter ms_handle_connect 0x7f239413c480 2024-03-25T19:17:54.743 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.743+0000 7f238a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:17:54.747 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.747+0000 7f238a7fc640 1 -- 172.21.15.6:0/2694774487 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1768424611 0 0) 0x7f2394144d60 con 0x7f23940b4ff0 2024-03-25T19:17:54.747 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.747+0000 7f238a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:17:54.747 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.747+0000 7f238a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:54.747 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.747+0000 7f238a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:54.747 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.747+0000 7f238a7fc640 10 monclient(hunting): my global_id is 4394 2024-03-25T19:17:54.747 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.747+0000 7f238a7fc640 1 -- 172.21.15.6:0/2694774487 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2394144a80 con 0x7f23940b4ff0 2024-03-25T19:17:54.754 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.754+0000 7f238a7fc640 1 -- 172.21.15.6:0/2694774487 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 232851058 0 0) 0x7f2378004eb0 con 0x7f23940b4670 2024-03-25T19:17:54.754 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.754+0000 7f238a7fc640 1 -- 172.21.15.6:0/2694774487 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2394144d60 con 0x7f23940b4670 2024-03-25T19:17:54.754 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.754+0000 7f238a7fc640 1 -- 172.21.15.6:0/2694774487 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f2390001990 con 0x7f23940b4670 2024-03-25T19:17:54.754 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.754+0000 7f238a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:17:54.754 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.754+0000 7f238a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:17:54.754 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.754+0000 7f238a7fc640 10 monclient(hunting): dump: 2024-03-25T19:17:54.754 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:17:54.754 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:54.754 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:54.754 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:54.754 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:17:54.755 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:54.755 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:17:54.755 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:17:54.755 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:17:54.755 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:54.755 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.754+0000 7f238a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:17:54.757 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.757+0000 7f238a7fc640 1 -- 172.21.15.6:0/2694774487 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2028285608 0 0) 0x7f2394144a80 con 0x7f23940b4ff0 2024-03-25T19:17:54.757 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.757+0000 7f238a7fc640 1 -- 172.21.15.6:0/2694774487 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2378004eb0 con 0x7f23940b4ff0 2024-03-25T19:17:54.757 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.757+0000 7f238a7fc640 1 -- 172.21.15.6:0/2694774487 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f23840018b0 con 0x7f23940b4ff0 2024-03-25T19:17:54.757 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.757+0000 7f238a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:17:54.757 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.757+0000 7f238a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:17:54.757 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.757+0000 7f238a7fc640 10 monclient(hunting): dump: 2024-03-25T19:17:54.757 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:17:54.757 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:54.757 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:54.757 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:54.757 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:17:54.758 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:54.758 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:17:54.758 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:17:54.758 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:17:54.758 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:54.759 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.759+0000 7f238a7fc640 1 -- 172.21.15.6:0/2694774487 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2482367138 0 0) 0x7f2394143d20 con 0x7f239413c480 2024-03-25T19:17:54.759 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.759+0000 7f238a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:17:54.759 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.759+0000 7f238a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:17:54.759 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.759+0000 7f238a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:17:54.759 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.759+0000 7f238a7fc640 10 monclient(hunting): my global_id is 4392 2024-03-25T19:17:54.759 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.759+0000 7f238a7fc640 1 -- 172.21.15.6:0/2694774487 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2394144a80 con 0x7f239413c480 2024-03-25T19:17:54.762 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.761+0000 7f238a7fc640 1 -- 172.21.15.6:0/2694774487 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2813573205 0 0) 0x7f2378004eb0 con 0x7f23940b4ff0 2024-03-25T19:17:54.762 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.761+0000 7f238a7fc640 1 -- 172.21.15.6:0/2694774487 >> v1:172.21.15.192:6789/0 conn(0x7f23940b4670 legacy=0x7f23941457d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:54.762 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.761+0000 7f238a7fc640 1 -- 172.21.15.6:0/2694774487 >> v1:172.21.15.6:6789/0 conn(0x7f239413c480 legacy=0x7f23941e2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:54.762 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.761+0000 7f238a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:17:54.762 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.761+0000 7f238a7fc640 1 monclient: found mon.c 2024-03-25T19:17:54.762 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.761+0000 7f238a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:17:54.762 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.761+0000 7f238a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:17:54.762 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.761+0000 7f238a7fc640 1 -- 172.21.15.6:0/2694774487 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2394145040 con 0x7f23940b4ff0 2024-03-25T19:17:54.762 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.761+0000 7f238a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:17:54.763 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.761+0000 7f238a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:17:54.763 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.761+0000 7f238a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:17:54.763 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.761+0000 7f239bc84640 5 monclient: authenticate success, global_id 4394 2024-03-25T19:17:54.763 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.761+0000 7f239bc84640 10 monclient: _renew_subs 2024-03-25T19:17:54.763 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.761+0000 7f239bc84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:17:54.763 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.761+0000 7f239bc84640 1 -- 172.21.15.6:0/2694774487 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2394143f90 con 0x7f23940b4ff0 2024-03-25T19:17:54.763 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.762+0000 7f239bc84640 10 client.4394.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:17:54.763 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.762+0000 7f239bc84640 10 monclient: _renew_subs 2024-03-25T19:17:54.763 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.762+0000 7f239bc84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:17:54.763 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.762+0000 7f239bc84640 1 -- 172.21.15.6:0/2694774487 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f239413dcb0 con 0x7f23940b4ff0 2024-03-25T19:17:54.763 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.762+0000 7f239bc84640 1 librados: init done 2024-03-25T19:17:54.764 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.763+0000 7f239bc84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:17:54.764 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.763+0000 7f239bc84640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:17:54.764 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.763+0000 7f239bc84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:17:54.764 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:54.763+0000 7f239bc84640 1 -- 172.21.15.6:0/2694774487 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f239413df60 con 0x7f23940b4ff0 2024-03-25T19:17:55.082 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:55.082+0000 7f238a7fc640 1 -- 172.21.15.6:0/2694774487 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2384001d70 con 0x7f23940b4ff0 2024-03-25T19:17:55.082 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:55.082+0000 7f238a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:17:55.083 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:55.082+0000 7f238a7fc640 1 -- 172.21.15.6:0/2694774487 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f2384004a00 con 0x7f23940b4ff0 2024-03-25T19:17:55.083 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:55.082+0000 7f238a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:17:55.083 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:55.082+0000 7f238a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:17:55.083 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:55.082+0000 7f238a7fc640 10 monclient: dump: 2024-03-25T19:17:55.083 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:17:55.083 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:17:55.083 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:55.083 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:17:55.083 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:17:55.083 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:17:55.083 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:17:55.083 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:17:55.083 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:17:55.083 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:17:55.083 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:55.082+0000 7f238bfff640 4 set_mon_vals no callback set 2024-03-25T19:17:55.084 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:55.083+0000 7f238a7fc640 1 -- 172.21.15.6:0/2694774487 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f2384004c20 con 0x7f23940b4ff0 2024-03-25T19:17:55.084 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:55.084+0000 7f2399a20640 10 monclient: get_auth_request con 0x7f237806dee0 auth_method 0 2024-03-25T19:17:55.084 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:55.084+0000 7f238a7fc640 10 client.4394.objecter ms_handle_connect 0x7f237806dee0 2024-03-25T19:17:55.089 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:55.088+0000 7f238a7fc640 1 -- 172.21.15.6:0/2694774487 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f2384054780 con 0x7f23940b4ff0 2024-03-25T19:17:55.089 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:55.088+0000 7f238a7fc640 10 client.4394.objecter ms_dispatch 0x7f2394000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:17:55.089 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:55.088+0000 7f238a7fc640 3 client.4394.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:17:55.089 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:55.088+0000 7f238a7fc640 3 client.4394.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:17:55.089 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:55.088+0000 7f238a7fc640 20 client.4394.objecter dump_active .. 0 homeless 2024-03-25T19:17:55.089 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:55.089+0000 7f238a7fc640 1 -- 172.21.15.6:0/2694774487 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2384050290 con 0x7f23940b4ff0 2024-03-25T19:17:55.089 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:55.089+0000 7f238a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:17:55.089 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:55.089+0000 7f238a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:17:55.345 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:55.345+0000 7f239bc84640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:17:55.345 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:55.345+0000 7f239bc84640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:17:55.346 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:55.345+0000 7f239bc84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:17:55.346 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:55.345+0000 7f239bc84640 1 -- 172.21.15.6:0/2694774487 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f239406dbd0 con 0x7f23940b4ff0 2024-03-25T19:17:55.379 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:55.379+0000 7f238a7fc640 1 -- 172.21.15.6:0/2694774487 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f23840521c0 con 0x7f23940b4ff0 2024-03-25T19:17:55.379 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:55.379+0000 7f238a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:17:55.379 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:55.379+0000 7f238a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:17:55.380 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:17:55.380 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:17:55.382 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:55.381+0000 7f239bc84640 10 librados: watch_flush enter 2024-03-25T19:17:55.382 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:55.381+0000 7f239bc84640 10 librados: watch_flush exit 2024-03-25T19:17:55.382 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:55.381+0000 7f239bc84640 20 client.4394.objecter shutdown clearing up homeless session... 2024-03-25T19:17:55.382 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:55.381+0000 7f239bc84640 10 client.4394.objecter successfully canceled tick 2024-03-25T19:17:55.382 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:55.381+0000 7f239bc84640 1 -- 172.21.15.6:0/2694774487 >> v1:172.21.15.6:6800/2936652650 conn(0x7f237806dee0 legacy=0x7f23780703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:55.382 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:55.381+0000 7f239bc84640 10 monclient: shutdown 2024-03-25T19:17:55.382 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:55.381+0000 7f239bc84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:17:55.382 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:55.381+0000 7f239bc84640 1 -- 172.21.15.6:0/2694774487 >> v1:172.21.15.6:6790/0 conn(0x7f23940b4ff0 legacy=0x7f2394145ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:17:55.382 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:55.381+0000 7f239bc84640 1 -- 172.21.15.6:0/2694774487 shutdown_connections 2024-03-25T19:17:55.382 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:55.381+0000 7f239bc84640 1 -- 172.21.15.6:0/2694774487 >> 172.21.15.6:0/2694774487 conn(0x7f23940617a0 msgr2=0x7f2394141ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:17:55.382 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:55.381+0000 7f239bc84640 1 -- 172.21.15.6:0/2694774487 shutdown_connections 2024-03-25T19:17:55.382 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:55.381+0000 7f239bc84640 1 -- 172.21.15.6:0/2694774487 wait complete. 2024-03-25T19:17:55.383 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:17:55.382+0000 7f239bc84640 1 librados: shutdown 2024-03-25T19:17:55.393 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:18:02.200 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:18:02.315 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.315+0000 7fa1e3d6c640 10 monclient: get_monmap_and_config 2024-03-25T19:18:02.315 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.315+0000 7fa1e3d6c640 10 monclient: build_initial_monmap 2024-03-25T19:18:02.315 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.315+0000 7fa1e3d6c640 10 monclient: monmap: 2024-03-25T19:18:02.315 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:18:02.315 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:02.315 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:18:02.316007+0000 2024-03-25T19:18:02.315 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:18:02.316007+0000 2024-03-25T19:18:02.315 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:18:02.315 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:02.315 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:18:02.315 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:18:02.316 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:18:02.316 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:02.317 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.315+0000 7fa1e3d6c640 1 Processor -- start 2024-03-25T19:18:02.317 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.315+0000 7fa1e3d6c640 1 -- start start 2024-03-25T19:18:02.317 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.315+0000 7fa1e3d6c640 10 monclient: init 2024-03-25T19:18:02.317 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.316+0000 7fa1e3d6c640 10 monclient: _reopen_session rank -1 2024-03-25T19:18:02.317 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.316+0000 7fa1e3d6c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:18:02.318 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.316+0000 7fa1e3d6c640 10 monclient(hunting): picked mon.noname-b con 0x7fa1dc0703b0 addr v1:172.21.15.6:6790/0 2024-03-25T19:18:02.318 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.316+0000 7fa1e3d6c640 10 monclient(hunting): picked mon.noname-a con 0x7fa1dc140ba0 addr v1:172.21.15.6:6789/0 2024-03-25T19:18:02.318 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.316+0000 7fa1e3d6c640 10 monclient(hunting): picked mon.noname-c con 0x7fa1dc0af6e0 addr v1:172.21.15.192:6789/0 2024-03-25T19:18:02.318 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.316+0000 7fa1e3d6c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1dc0723b0 con 0x7fa1dc140ba0 2024-03-25T19:18:02.318 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.316+0000 7fa1e3d6c640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1dc141020 con 0x7fa1dc0af6e0 2024-03-25T19:18:02.318 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.317+0000 7fa1e3d6c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1dc072600 con 0x7fa1dc0703b0 2024-03-25T19:18:02.318 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.317+0000 7fa1e3d6c640 10 monclient(hunting): _renew_subs 2024-03-25T19:18:02.318 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.317+0000 7fa1e3d6c640 10 monclient(hunting): authenticate will time out at 6186.556913s 2024-03-25T19:18:02.323 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.323+0000 7fa1e2309640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fa1dc0af6e0 0x7fa1dc14b9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:34206/0 (socket says 172.21.15.6:34206) 2024-03-25T19:18:02.323 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.323+0000 7fa1e2309640 1 -- 172.21.15.6:0/564487252 learned_addr learned my addr 172.21.15.6:0/564487252 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:18:02.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.342+0000 7fa1e0b06640 1 -- 172.21.15.6:0/564487252 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1802156355 0 0) 0x7fa1dc141020 con 0x7fa1dc0af6e0 2024-03-25T19:18:02.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.342+0000 7fa1e0b06640 10 monclient(hunting): _negotiate 2024-03-25T19:18:02.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.342+0000 7fa1e0b06640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:02.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.342+0000 7fa1e0b06640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:02.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.342+0000 7fa1e0b06640 10 monclient(hunting): my global_id is 4399 2024-03-25T19:18:02.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.342+0000 7fa1e0b06640 1 -- 172.21.15.6:0/564487252 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1c8003520 con 0x7fa1dc0af6e0 2024-03-25T19:18:02.345 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.345+0000 7fa1e0b06640 1 -- 172.21.15.6:0/564487252 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3608326814 0 0) 0x7fa1dc072600 con 0x7fa1dc0703b0 2024-03-25T19:18:02.345 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.345+0000 7fa1e0b06640 10 monclient(hunting): _negotiate 2024-03-25T19:18:02.345 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.345+0000 7fa1e0b06640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:02.346 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.345+0000 7fa1e0b06640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:02.346 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.345+0000 7fa1e0b06640 10 monclient(hunting): my global_id is 4397 2024-03-25T19:18:02.346 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.345+0000 7fa1e0b06640 1 -- 172.21.15.6:0/564487252 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1dc141020 con 0x7fa1dc0703b0 2024-03-25T19:18:02.358 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.357+0000 7fa1e0b06640 1 -- 172.21.15.6:0/564487252 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 940462190 0 0) 0x7fa1dc0723b0 con 0x7fa1dc140ba0 2024-03-25T19:18:02.358 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.357+0000 7fa1e0b06640 10 monclient(hunting): _negotiate 2024-03-25T19:18:02.358 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.357+0000 7fa1e0b06640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:02.358 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.357+0000 7fa1e0b06640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:02.358 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.357+0000 7fa1e0b06640 10 monclient(hunting): my global_id is 4395 2024-03-25T19:18:02.358 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.357+0000 7fa1e0b06640 1 -- 172.21.15.6:0/564487252 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1dc072600 con 0x7fa1dc140ba0 2024-03-25T19:18:02.361 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.361+0000 7fa1e0b06640 1 -- 172.21.15.6:0/564487252 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 58604640 0 0) 0x7fa1dc141020 con 0x7fa1dc0703b0 2024-03-25T19:18:02.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.361+0000 7fa1e0b06640 1 -- 172.21.15.6:0/564487252 >> v1:172.21.15.192:6789/0 conn(0x7fa1dc0af6e0 legacy=0x7fa1dc14b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:02.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.361+0000 7fa1e0b06640 1 -- 172.21.15.6:0/564487252 >> v1:172.21.15.6:6789/0 conn(0x7fa1dc140ba0 legacy=0x7fa1dc139cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:02.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.361+0000 7fa1e0b06640 10 monclient: _finish_hunting 0 2024-03-25T19:18:02.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.361+0000 7fa1e0b06640 1 monclient: found mon.noname-b 2024-03-25T19:18:02.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.361+0000 7fa1e0b06640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:18:02.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.361+0000 7fa1e0b06640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-03-25T19:18:02.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.361+0000 7fa1e0b06640 1 -- 172.21.15.6:0/564487252 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1dc0728e0 con 0x7fa1dc0703b0 2024-03-25T19:18:02.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.361+0000 7fa1e0b06640 10 monclient: _finish_auth 0 2024-03-25T19:18:02.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.361+0000 7fa1e0b06640 10 monclient: _check_auth_tickets 2024-03-25T19:18:02.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.361+0000 7fa1e0b06640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:18:02.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.361+0000 7fa1e3d6c640 5 monclient: authenticate success, global_id 4397 2024-03-25T19:18:02.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.361+0000 7fa1e0b06640 1 -- 172.21.15.6:0/564487252 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fa1d8001670 con 0x7fa1dc0703b0 2024-03-25T19:18:02.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.361+0000 7fa1e0b06640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:18:02.363 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.361+0000 7fa1e0b06640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:18:02.363 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.361+0000 7fa1e0b06640 10 monclient: dump: 2024-03-25T19:18:02.363 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:18:02.363 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:02.363 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:02.363 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:02.363 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:18:02.363 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:02.363 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:18:02.363 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:18:02.363 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:18:02.363 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:02.363 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.361+0000 7fa1e3d6c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:18:02.387 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.387+0000 7fa1e0b06640 1 -- 172.21.15.6:0/564487252 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1d80017d0 con 0x7fa1dc0703b0 2024-03-25T19:18:02.387 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.387+0000 7fa1e0b06640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:18:02.388 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.387+0000 7fa1e0b06640 1 -- 172.21.15.6:0/564487252 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fa1d8001b60 con 0x7fa1dc0703b0 2024-03-25T19:18:02.388 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.387+0000 7fa1e3d6c640 10 monclient: get_monmap_and_config success 2024-03-25T19:18:02.388 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.387+0000 7fa1e3d6c640 4 set_mon_vals no callback set 2024-03-25T19:18:02.388 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.387+0000 7fa1e0b06640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:18:02.388 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.387+0000 7fa1e0b06640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:18:02.388 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.387+0000 7fa1e0b06640 10 monclient: dump: 2024-03-25T19:18:02.388 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:18:02.389 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:02.389 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:02.389 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:02.389 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:18:02.389 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:02.389 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:18:02.390 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:18:02.390 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:18:02.390 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:02.393 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.388+0000 7fa1e3d6c640 10 monclient: shutdown 2024-03-25T19:18:02.393 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.388+0000 7fa1e3d6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:18:02.394 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.388+0000 7fa1e3d6c640 1 -- 172.21.15.6:0/564487252 >> v1:172.21.15.6:6790/0 conn(0x7fa1dc0703b0 legacy=0x7fa1dc070790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:02.394 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.388+0000 7fa1e3d6c640 1 -- 172.21.15.6:0/564487252 shutdown_connections 2024-03-25T19:18:02.394 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.388+0000 7fa1e3d6c640 1 -- 172.21.15.6:0/564487252 >> 172.21.15.6:0/564487252 conn(0x7fa1dc0b3cd0 msgr2=0x7fa1dc0b40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:18:02.394 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.388+0000 7fa1e3d6c640 1 -- 172.21.15.6:0/564487252 shutdown_connections 2024-03-25T19:18:02.394 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.388+0000 7fa1e3d6c640 1 -- 172.21.15.6:0/564487252 wait complete. 2024-03-25T19:18:02.394 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.389+0000 7fa1e3d6c640 10 monclient: build_initial_monmap 2024-03-25T19:18:02.394 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.389+0000 7fa1e3d6c640 10 monclient: monmap: 2024-03-25T19:18:02.395 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:18:02.395 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:18:02.395 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:18:02.395 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:18:02.395 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:18:02.395 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:18:02.395 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:18:02.395 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:18:02.395 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:18:02.395 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:02.395 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.389+0000 7fa1e3d6c640 1 librados: starting msgr at 2024-03-25T19:18:02.395 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.389+0000 7fa1e3d6c640 1 librados: starting objecter 2024-03-25T19:18:02.396 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.389+0000 7fa1e3d6c640 1 Processor -- start 2024-03-25T19:18:02.396 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.389+0000 7fa1e3d6c640 1 -- start start 2024-03-25T19:18:02.396 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.389+0000 7fa1e3d6c640 1 librados: setting wanted keys 2024-03-25T19:18:02.396 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.389+0000 7fa1e3d6c640 1 librados: calling monclient init 2024-03-25T19:18:02.396 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.389+0000 7fa1e3d6c640 10 monclient: init 2024-03-25T19:18:02.396 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.390+0000 7fa1e3d6c640 10 monclient: _reopen_session rank -1 2024-03-25T19:18:02.396 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.390+0000 7fa1e3d6c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:18:02.396 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.390+0000 7fa1e3d6c640 10 monclient(hunting): picked mon.noname-b con 0x7fa1dc0703b0 addr v1:172.21.15.192:6789/0 2024-03-25T19:18:02.396 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.390+0000 7fa1e3d6c640 10 monclient(hunting): picked mon.noname-c con 0x7fa1dc0af6e0 addr v1:172.21.15.6:6790/0 2024-03-25T19:18:02.396 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.390+0000 7fa1e3d6c640 10 monclient(hunting): picked mon.noname-a con 0x7fa1dc140ba0 addr v1:172.21.15.6:6789/0 2024-03-25T19:18:02.396 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.390+0000 7fa1e3d6c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1dc1d7450 con 0x7fa1dc140ba0 2024-03-25T19:18:02.396 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.390+0000 7fa1e3d6c640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1dc1d70e0 con 0x7fa1dc0703b0 2024-03-25T19:18:02.396 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.390+0000 7fa1e3d6c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1dc11cf00 con 0x7fa1dc0af6e0 2024-03-25T19:18:02.396 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.390+0000 7fa1e3d6c640 10 monclient(hunting): _renew_subs 2024-03-25T19:18:02.396 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.390+0000 7fa1e3d6c640 10 monclient(hunting): authenticate will time out at 6186.630371s 2024-03-25T19:18:02.398 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.399+0000 7fa1e1b08640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fa1dc0703b0 0x7fa1dc1d7970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:34284/0 (socket says 172.21.15.6:34284) 2024-03-25T19:18:02.399 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.399+0000 7fa1e1b08640 1 -- 172.21.15.6:0/2982461768 learned_addr learned my addr 172.21.15.6:0/2982461768 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:18:02.399 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.400+0000 7fa1d27fc640 10 client.?.objecter ms_handle_connect 0x7fa1dc0703b0 2024-03-25T19:18:02.399 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.400+0000 7fa1d27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:18:02.405 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.405+0000 7fa1d27fc640 10 client.?.objecter ms_handle_connect 0x7fa1dc0af6e0 2024-03-25T19:18:02.405 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.405+0000 7fa1d27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:18:02.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.408+0000 7fa1d27fc640 10 client.?.objecter ms_handle_connect 0x7fa1dc140ba0 2024-03-25T19:18:02.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.408+0000 7fa1d27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:18:02.411 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.411+0000 7fa1d27fc640 1 -- 172.21.15.6:0/2982461768 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3260991351 0 0) 0x7fa1dc1d70e0 con 0x7fa1dc0703b0 2024-03-25T19:18:02.411 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.411+0000 7fa1d27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:18:02.411 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.411+0000 7fa1d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:02.411 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.411+0000 7fa1d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:02.411 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.411+0000 7fa1d27fc640 10 monclient(hunting): my global_id is 4402 2024-03-25T19:18:02.411 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.411+0000 7fa1d27fc640 1 -- 172.21.15.6:0/2982461768 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1c0004eb0 con 0x7fa1dc0703b0 2024-03-25T19:18:02.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.416+0000 7fa1d27fc640 1 -- 172.21.15.6:0/2982461768 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2791803445 0 0) 0x7fa1dc11cf00 con 0x7fa1dc0af6e0 2024-03-25T19:18:02.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.416+0000 7fa1d27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:18:02.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.416+0000 7fa1d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:02.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.416+0000 7fa1d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:02.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.416+0000 7fa1d27fc640 10 monclient(hunting): my global_id is 4400 2024-03-25T19:18:02.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.416+0000 7fa1d27fc640 1 -- 172.21.15.6:0/2982461768 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1dc1d70e0 con 0x7fa1dc0af6e0 2024-03-25T19:18:02.426 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.426+0000 7fa1d27fc640 1 -- 172.21.15.6:0/2982461768 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3765234281 0 0) 0x7fa1dc1d7450 con 0x7fa1dc140ba0 2024-03-25T19:18:02.426 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.426+0000 7fa1d27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:18:02.427 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.426+0000 7fa1d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:02.427 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.426+0000 7fa1d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:02.427 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.426+0000 7fa1d27fc640 10 monclient(hunting): my global_id is 4398 2024-03-25T19:18:02.427 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.426+0000 7fa1d27fc640 1 -- 172.21.15.6:0/2982461768 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1dc11cf00 con 0x7fa1dc140ba0 2024-03-25T19:18:02.432 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.431+0000 7fa1d27fc640 1 -- 172.21.15.6:0/2982461768 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3619523115 0 0) 0x7fa1c0004eb0 con 0x7fa1dc0703b0 2024-03-25T19:18:02.432 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.431+0000 7fa1d27fc640 1 -- 172.21.15.6:0/2982461768 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1dc1d7450 con 0x7fa1dc0703b0 2024-03-25T19:18:02.432 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.431+0000 7fa1d27fc640 1 -- 172.21.15.6:0/2982461768 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fa1d8001c20 con 0x7fa1dc0703b0 2024-03-25T19:18:02.432 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.431+0000 7fa1d27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:18:02.432 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.431+0000 7fa1d27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:18:02.432 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.431+0000 7fa1d27fc640 10 monclient(hunting): dump: 2024-03-25T19:18:02.432 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:18:02.432 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:02.432 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:02.432 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:02.432 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:18:02.432 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:02.432 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:18:02.433 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:18:02.433 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:18:02.433 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:02.433 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.431+0000 7fa1d27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:18:02.439 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.439+0000 7fa1d27fc640 1 -- 172.21.15.6:0/2982461768 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2764538053 0 0) 0x7fa1dc1d70e0 con 0x7fa1dc0af6e0 2024-03-25T19:18:02.439 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.439+0000 7fa1d27fc640 1 -- 172.21.15.6:0/2982461768 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1c0004eb0 con 0x7fa1dc0af6e0 2024-03-25T19:18:02.440 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.439+0000 7fa1d27fc640 1 -- 172.21.15.6:0/2982461768 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fa1cc001870 con 0x7fa1dc0af6e0 2024-03-25T19:18:02.440 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.439+0000 7fa1d27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:18:02.440 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.439+0000 7fa1d27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:18:02.440 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.439+0000 7fa1d27fc640 10 monclient(hunting): dump: 2024-03-25T19:18:02.440 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:18:02.440 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:02.440 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:02.440 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:02.440 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:18:02.440 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:02.440 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:18:02.440 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:18:02.440 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:18:02.440 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:02.442 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.441+0000 7fa1d27fc640 1 -- 172.21.15.6:0/2982461768 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 547083094 0 0) 0x7fa1dc1d7450 con 0x7fa1dc0703b0 2024-03-25T19:18:02.442 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.441+0000 7fa1d27fc640 1 -- 172.21.15.6:0/2982461768 >> v1:172.21.15.6:6790/0 conn(0x7fa1dc0af6e0 legacy=0x7fa1dc11c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:02.442 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.441+0000 7fa1d27fc640 1 -- 172.21.15.6:0/2982461768 >> v1:172.21.15.6:6789/0 conn(0x7fa1dc140ba0 legacy=0x7fa1dc138e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:02.442 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.441+0000 7fa1d27fc640 10 monclient: _finish_hunting 0 2024-03-25T19:18:02.442 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.441+0000 7fa1d27fc640 1 monclient: found mon.b 2024-03-25T19:18:02.442 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.441+0000 7fa1d27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:18:02.442 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.441+0000 7fa1d27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:18:02.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.441+0000 7fa1d27fc640 1 -- 172.21.15.6:0/2982461768 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1dc11d090 con 0x7fa1dc0703b0 2024-03-25T19:18:02.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.441+0000 7fa1d27fc640 10 monclient: _finish_auth 0 2024-03-25T19:18:02.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.441+0000 7fa1d27fc640 10 monclient: _check_auth_tickets 2024-03-25T19:18:02.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.441+0000 7fa1d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:18:02.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.441+0000 7fa1e3d6c640 5 monclient: authenticate success, global_id 4402 2024-03-25T19:18:02.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.441+0000 7fa1e3d6c640 10 monclient: _renew_subs 2024-03-25T19:18:02.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.441+0000 7fa1e3d6c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:18:02.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.441+0000 7fa1e3d6c640 1 -- 172.21.15.6:0/2982461768 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1dc139560 con 0x7fa1dc0703b0 2024-03-25T19:18:02.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.442+0000 7fa1e3d6c640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:18:02.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.442+0000 7fa1e3d6c640 10 monclient: _renew_subs 2024-03-25T19:18:02.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.442+0000 7fa1e3d6c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:18:02.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.442+0000 7fa1e3d6c640 1 -- 172.21.15.6:0/2982461768 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1dc11bcb0 con 0x7fa1dc0703b0 2024-03-25T19:18:02.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.442+0000 7fa1e3d6c640 1 librados: init done 2024-03-25T19:18:02.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.443+0000 7fa1e3d6c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:18:02.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.443+0000 7fa1e3d6c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:18:02.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.443+0000 7fa1e3d6c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:18:02.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.443+0000 7fa1e3d6c640 1 -- 172.21.15.6:0/2982461768 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1dc1d7270 con 0x7fa1dc0703b0 2024-03-25T19:18:02.466 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.465+0000 7fa1d27fc640 1 -- 172.21.15.6:0/2982461768 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1d80038a0 con 0x7fa1dc0703b0 2024-03-25T19:18:02.466 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.465+0000 7fa1d27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:18:02.466 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.465+0000 7fa1d27fc640 1 -- 172.21.15.6:0/2982461768 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fa1d8003c30 con 0x7fa1dc0703b0 2024-03-25T19:18:02.466 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.465+0000 7fa1d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:18:02.466 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.465+0000 7fa1d27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:18:02.466 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.465+0000 7fa1d27fc640 10 monclient: dump: 2024-03-25T19:18:02.466 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:18:02.466 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:02.466 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:02.466 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:02.467 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:18:02.467 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:02.467 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:18:02.467 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:18:02.467 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:18:02.467 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:02.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.465+0000 7fa1d3fff640 4 set_mon_vals no callback set 2024-03-25T19:18:02.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.467+0000 7fa1d27fc640 1 -- 172.21.15.6:0/2982461768 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7fa1d8003d90 con 0x7fa1dc0703b0 2024-03-25T19:18:02.468 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.467+0000 7fa1e1307640 10 monclient: get_auth_request con 0x7fa1c006df70 auth_method 0 2024-03-25T19:18:02.468 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.467+0000 7fa1d27fc640 10 client.4402.objecter ms_handle_connect 0x7fa1c006df70 2024-03-25T19:18:02.719 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.719+0000 7fa1d27fc640 1 -- 172.21.15.6:0/2982461768 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7fa1d80548d0 con 0x7fa1dc0703b0 2024-03-25T19:18:02.719 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.719+0000 7fa1d27fc640 10 client.4402.objecter ms_dispatch 0x7fa1dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:18:02.719 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.719+0000 7fa1d27fc640 3 client.4402.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:18:02.719 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.719+0000 7fa1d27fc640 3 client.4402.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:18:02.719 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.719+0000 7fa1d27fc640 20 client.4402.objecter dump_active .. 0 homeless 2024-03-25T19:18:02.719 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.719+0000 7fa1d27fc640 1 -- 172.21.15.6:0/2982461768 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa1d80503e0 con 0x7fa1dc0703b0 2024-03-25T19:18:02.720 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.719+0000 7fa1d27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:18:02.720 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.719+0000 7fa1d27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:18:02.951 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.951+0000 7fa1e3d6c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:18:02.951 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.951+0000 7fa1e3d6c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:18:02.951 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.951+0000 7fa1e3d6c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:18:02.952 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:02.951+0000 7fa1e3d6c640 1 -- 172.21.15.6:0/2982461768 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1dc1d75e0 con 0x7fa1dc0703b0 2024-03-25T19:18:03.096 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:03.096+0000 7fa1d27fc640 1 -- 172.21.15.6:0/2982461768 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7fa1d8052310 con 0x7fa1dc0703b0 2024-03-25T19:18:03.096 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:03.096+0000 7fa1d27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:18:03.096 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:03.096+0000 7fa1d27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:18:03.096 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:18:03.097 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:18:03.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:03.098+0000 7fa1e3d6c640 10 librados: watch_flush enter 2024-03-25T19:18:03.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:03.098+0000 7fa1e3d6c640 10 librados: watch_flush exit 2024-03-25T19:18:03.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:03.098+0000 7fa1e3d6c640 20 client.4402.objecter shutdown clearing up homeless session... 2024-03-25T19:18:03.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:03.098+0000 7fa1e3d6c640 10 client.4402.objecter successfully canceled tick 2024-03-25T19:18:03.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:03.098+0000 7fa1e3d6c640 1 -- 172.21.15.6:0/2982461768 >> v1:172.21.15.6:6800/2936652650 conn(0x7fa1c006df70 legacy=0x7fa1c0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:03.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:03.098+0000 7fa1e3d6c640 10 monclient: shutdown 2024-03-25T19:18:03.100 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:03.098+0000 7fa1e3d6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:18:03.100 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:03.098+0000 7fa1e3d6c640 1 -- 172.21.15.6:0/2982461768 >> v1:172.21.15.192:6789/0 conn(0x7fa1dc0703b0 legacy=0x7fa1dc1d7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:03.100 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:03.099+0000 7fa1e3d6c640 1 -- 172.21.15.6:0/2982461768 shutdown_connections 2024-03-25T19:18:03.100 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:03.099+0000 7fa1e3d6c640 1 -- 172.21.15.6:0/2982461768 >> 172.21.15.6:0/2982461768 conn(0x7fa1dc0b3cd0 msgr2=0x7fa1dc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:18:03.100 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:03.099+0000 7fa1e3d6c640 1 -- 172.21.15.6:0/2982461768 shutdown_connections 2024-03-25T19:18:03.100 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:03.099+0000 7fa1e3d6c640 1 -- 172.21.15.6:0/2982461768 wait complete. 2024-03-25T19:18:03.100 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:03.099+0000 7fa1e3d6c640 1 librados: shutdown 2024-03-25T19:18:03.111 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:18:09.917 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:18:10.035 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.035+0000 7f5bf904e640 10 monclient: get_monmap_and_config 2024-03-25T19:18:10.036 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.035+0000 7f5bf904e640 10 monclient: build_initial_monmap 2024-03-25T19:18:10.036 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.035+0000 7f5bf904e640 10 monclient: monmap: 2024-03-25T19:18:10.036 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:18:10.036 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:10.036 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:18:10.036812+0000 2024-03-25T19:18:10.037 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:18:10.036812+0000 2024-03-25T19:18:10.037 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:18:10.037 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:10.037 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:18:10.037 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:18:10.038 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:18:10.038 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:10.039 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.036+0000 7f5bf904e640 1 Processor -- start 2024-03-25T19:18:10.039 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.036+0000 7f5bf904e640 1 -- start start 2024-03-25T19:18:10.039 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.036+0000 7f5bf904e640 10 monclient: init 2024-03-25T19:18:10.039 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.037+0000 7f5bf904e640 10 monclient: _reopen_session rank -1 2024-03-25T19:18:10.039 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.037+0000 7f5bf904e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:18:10.039 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.037+0000 7f5bf904e640 10 monclient(hunting): picked mon.noname-c con 0x7f5bf40b4e70 addr v1:172.21.15.192:6789/0 2024-03-25T19:18:10.040 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.037+0000 7f5bf904e640 10 monclient(hunting): picked mon.noname-a con 0x7f5bf40b44b0 addr v1:172.21.15.6:6789/0 2024-03-25T19:18:10.040 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.037+0000 7f5bf904e640 10 monclient(hunting): picked mon.noname-b con 0x7f5bf40b1b00 addr v1:172.21.15.6:6790/0 2024-03-25T19:18:10.040 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.037+0000 7f5bf904e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bf40a96f0 con 0x7f5bf40b44b0 2024-03-25T19:18:10.040 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.037+0000 7f5bf904e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bf40ab0e0 con 0x7f5bf40b4e70 2024-03-25T19:18:10.040 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.037+0000 7f5bf904e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bf4145f70 con 0x7f5bf40b1b00 2024-03-25T19:18:10.040 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.037+0000 7f5bf904e640 10 monclient(hunting): _renew_subs 2024-03-25T19:18:10.040 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.037+0000 7f5bf904e640 10 monclient(hunting): authenticate will time out at 6194.277459s 2024-03-25T19:18:10.050 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.051+0000 7f5bf259c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5bf40b44b0 0x7f5bf4145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53658/0 (socket says 172.21.15.6:53658) 2024-03-25T19:18:10.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.051+0000 7f5bf259c640 1 -- 172.21.15.6:0/1718246786 learned_addr learned my addr 172.21.15.6:0/1718246786 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:18:10.067 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.068+0000 7f5bf1d9b640 1 -- 172.21.15.6:0/1718246786 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1161115942 0 0) 0x7f5bf40ab0e0 con 0x7f5bf40b4e70 2024-03-25T19:18:10.068 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.068+0000 7f5bf1d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:18:10.068 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.068+0000 7f5bf1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:10.068 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.068+0000 7f5bf1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:10.068 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.068+0000 7f5bf1d9b640 10 monclient(hunting): my global_id is 4405 2024-03-25T19:18:10.068 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.068+0000 7f5bf1d9b640 1 -- 172.21.15.6:0/1718246786 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bdc003520 con 0x7f5bf40b4e70 2024-03-25T19:18:10.077 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.077+0000 7f5bf1d9b640 1 -- 172.21.15.6:0/1718246786 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1545094338 0 0) 0x7f5bf4145f70 con 0x7f5bf40b1b00 2024-03-25T19:18:10.077 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.077+0000 7f5bf1d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:18:10.077 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.077+0000 7f5bf1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:10.077 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.077+0000 7f5bf1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:10.078 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.077+0000 7f5bf1d9b640 10 monclient(hunting): my global_id is 4403 2024-03-25T19:18:10.078 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.077+0000 7f5bf1d9b640 1 -- 172.21.15.6:0/1718246786 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bf40ab0e0 con 0x7f5bf40b1b00 2024-03-25T19:18:10.079 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.078+0000 7f5bf1d9b640 1 -- 172.21.15.6:0/1718246786 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4159560048 0 0) 0x7f5bdc003520 con 0x7f5bf40b4e70 2024-03-25T19:18:10.079 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.078+0000 7f5bf1d9b640 1 -- 172.21.15.6:0/1718246786 >> v1:172.21.15.6:6790/0 conn(0x7f5bf40b1b00 legacy=0x7f5bf414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:10.079 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.078+0000 7f5bf1d9b640 1 -- 172.21.15.6:0/1718246786 >> v1:172.21.15.6:6789/0 conn(0x7f5bf40b44b0 legacy=0x7f5bf4145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:10.079 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.078+0000 7f5bf1d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:18:10.079 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.078+0000 7f5bf1d9b640 1 monclient: found mon.noname-c 2024-03-25T19:18:10.080 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.078+0000 7f5bf1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:18:10.080 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.078+0000 7f5bf1d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:18:10.080 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.078+0000 7f5bf1d9b640 1 -- 172.21.15.6:0/1718246786 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bf40a9b80 con 0x7f5bf40b4e70 2024-03-25T19:18:10.080 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.078+0000 7f5bf1d9b640 10 monclient: _finish_auth 0 2024-03-25T19:18:10.080 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.078+0000 7f5bf1d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:18:10.080 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.078+0000 7f5bf1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:18:10.080 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.078+0000 7f5bf904e640 5 monclient: authenticate success, global_id 4405 2024-03-25T19:18:10.080 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.078+0000 7f5bf904e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:18:10.080 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.078+0000 7f5bf1d9b640 1 -- 172.21.15.6:0/1718246786 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f5be4001690 con 0x7f5bf40b4e70 2024-03-25T19:18:10.080 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.078+0000 7f5bf1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:18:10.080 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.078+0000 7f5bf1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:18:10.081 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.078+0000 7f5bf1d9b640 10 monclient: dump: 2024-03-25T19:18:10.081 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:18:10.081 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:10.081 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:10.081 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:10.081 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:18:10.081 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:10.081 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:18:10.081 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:18:10.081 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:18:10.081 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:10.081 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.078+0000 7f5bf904e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:18:10.084 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.084+0000 7f5bf1d9b640 1 -- 172.21.15.6:0/1718246786 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5be40017f0 con 0x7f5bf40b4e70 2024-03-25T19:18:10.084 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.084+0000 7f5bf1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:18:10.085 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.084+0000 7f5bf1d9b640 1 -- 172.21.15.6:0/1718246786 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f5be4001c00 con 0x7f5bf40b4e70 2024-03-25T19:18:10.085 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.084+0000 7f5bf1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:18:10.085 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.084+0000 7f5bf1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:18:10.085 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.084+0000 7f5bf1d9b640 10 monclient: dump: 2024-03-25T19:18:10.085 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:18:10.085 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:10.085 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:10.085 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:10.085 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:18:10.085 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:10.085 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:18:10.085 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:18:10.086 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:18:10.086 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:10.086 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.084+0000 7f5bf904e640 10 monclient: get_monmap_and_config success 2024-03-25T19:18:10.086 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.084+0000 7f5bf904e640 4 set_mon_vals no callback set 2024-03-25T19:18:10.087 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.085+0000 7f5bf904e640 10 monclient: shutdown 2024-03-25T19:18:10.087 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.085+0000 7f5bf904e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:18:10.087 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.085+0000 7f5bf904e640 1 -- 172.21.15.6:0/1718246786 >> v1:172.21.15.192:6789/0 conn(0x7f5bf40b4e70 legacy=0x7f5bf413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:10.087 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.085+0000 7f5bf904e640 1 -- 172.21.15.6:0/1718246786 shutdown_connections 2024-03-25T19:18:10.088 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.085+0000 7f5bf904e640 1 -- 172.21.15.6:0/1718246786 >> 172.21.15.6:0/1718246786 conn(0x7f5bf40617a0 msgr2=0x7f5bf4061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:18:10.088 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.086+0000 7f5bf904e640 1 -- 172.21.15.6:0/1718246786 shutdown_connections 2024-03-25T19:18:10.088 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.086+0000 7f5bf904e640 1 -- 172.21.15.6:0/1718246786 wait complete. 2024-03-25T19:18:10.088 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.087+0000 7f5bf904e640 10 monclient: build_initial_monmap 2024-03-25T19:18:10.088 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.087+0000 7f5bf904e640 10 monclient: monmap: 2024-03-25T19:18:10.088 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:18:10.088 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:18:10.088 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:18:10.088 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:18:10.088 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:18:10.088 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:18:10.088 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:18:10.088 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:18:10.088 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:18:10.088 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:10.089 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.087+0000 7f5bf904e640 1 librados: starting msgr at 2024-03-25T19:18:10.089 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.087+0000 7f5bf904e640 1 librados: starting objecter 2024-03-25T19:18:10.089 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.087+0000 7f5bf904e640 1 Processor -- start 2024-03-25T19:18:10.089 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.087+0000 7f5bf904e640 1 -- start start 2024-03-25T19:18:10.089 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.087+0000 7f5bf904e640 1 librados: setting wanted keys 2024-03-25T19:18:10.089 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.087+0000 7f5bf904e640 1 librados: calling monclient init 2024-03-25T19:18:10.089 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.087+0000 7f5bf904e640 10 monclient: init 2024-03-25T19:18:10.090 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.088+0000 7f5bf904e640 10 monclient: _reopen_session rank -1 2024-03-25T19:18:10.090 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.088+0000 7f5bf904e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:18:10.090 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.088+0000 7f5bf904e640 10 monclient(hunting): picked mon.noname-a con 0x7f5bf40b1b00 addr v1:172.21.15.6:6789/0 2024-03-25T19:18:10.090 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.088+0000 7f5bf904e640 10 monclient(hunting): picked mon.noname-c con 0x7f5bf40b44b0 addr v1:172.21.15.6:6790/0 2024-03-25T19:18:10.090 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.088+0000 7f5bf904e640 10 monclient(hunting): picked mon.noname-b con 0x7f5bf40b4e70 addr v1:172.21.15.192:6789/0 2024-03-25T19:18:10.090 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.088+0000 7f5bf904e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bf413b940 con 0x7f5bf40b1b00 2024-03-25T19:18:10.091 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.088+0000 7f5bf904e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bf413cae0 con 0x7f5bf40b4e70 2024-03-25T19:18:10.091 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.088+0000 7f5bf904e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bf41dad30 con 0x7f5bf40b44b0 2024-03-25T19:18:10.091 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.088+0000 7f5bf904e640 10 monclient(hunting): _renew_subs 2024-03-25T19:18:10.091 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.088+0000 7f5bf904e640 10 monclient(hunting): authenticate will time out at 6194.328741s 2024-03-25T19:18:10.107 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.107+0000 7f5bf359e640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f5bf40b4e70 0x7f5bf41e2000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:43768/0 (socket says 172.21.15.6:43768) 2024-03-25T19:18:10.107 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.107+0000 7f5bf359e640 1 -- 172.21.15.6:0/2173318909 learned_addr learned my addr 172.21.15.6:0/2173318909 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:18:10.111 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.111+0000 7f5bdb7fe640 10 client.?.objecter ms_handle_connect 0x7f5bf40b4e70 2024-03-25T19:18:10.111 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.111+0000 7f5bdb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:18:10.113 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.113+0000 7f5bdb7fe640 10 client.?.objecter ms_handle_connect 0x7f5bf40b44b0 2024-03-25T19:18:10.113 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.113+0000 7f5bdb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:18:10.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.120+0000 7f5bdb7fe640 10 client.?.objecter ms_handle_connect 0x7f5bf40b1b00 2024-03-25T19:18:10.120 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.120+0000 7f5bdb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:18:10.121 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.121+0000 7f5bdb7fe640 1 -- 172.21.15.6:0/2173318909 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 836414935 0 0) 0x7f5bf413cae0 con 0x7f5bf40b4e70 2024-03-25T19:18:10.121 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.121+0000 7f5bdb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:18:10.122 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.121+0000 7f5bdb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:10.122 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.121+0000 7f5bdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:10.122 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.121+0000 7f5bdb7fe640 10 monclient(hunting): my global_id is 4408 2024-03-25T19:18:10.122 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.121+0000 7f5bdb7fe640 1 -- 172.21.15.6:0/2173318909 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5be0004eb0 con 0x7f5bf40b4e70 2024-03-25T19:18:10.122 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.121+0000 7f5bdb7fe640 1 -- 172.21.15.6:0/2173318909 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 214052791 0 0) 0x7f5bf41dad30 con 0x7f5bf40b44b0 2024-03-25T19:18:10.122 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.121+0000 7f5bdb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:18:10.122 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.121+0000 7f5bdb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:10.122 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.121+0000 7f5bdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:10.122 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.121+0000 7f5bdb7fe640 10 monclient(hunting): my global_id is 4406 2024-03-25T19:18:10.122 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.121+0000 7f5bdb7fe640 1 -- 172.21.15.6:0/2173318909 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bf413cae0 con 0x7f5bf40b44b0 2024-03-25T19:18:10.130 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.130+0000 7f5bdb7fe640 1 -- 172.21.15.6:0/2173318909 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2194154224 0 0) 0x7f5bf413b940 con 0x7f5bf40b1b00 2024-03-25T19:18:10.130 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.130+0000 7f5bdb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:18:10.130 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.130+0000 7f5bdb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:10.130 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.130+0000 7f5bdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:10.130 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.130+0000 7f5bdb7fe640 10 monclient(hunting): my global_id is 4404 2024-03-25T19:18:10.130 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.130+0000 7f5bdb7fe640 1 -- 172.21.15.6:0/2173318909 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bf41dad30 con 0x7f5bf40b1b00 2024-03-25T19:18:10.144 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.144+0000 7f5bdb7fe640 1 -- 172.21.15.6:0/2173318909 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3532086670 0 0) 0x7f5be0004eb0 con 0x7f5bf40b4e70 2024-03-25T19:18:10.144 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.144+0000 7f5bdb7fe640 1 -- 172.21.15.6:0/2173318909 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5bf413b940 con 0x7f5bf40b4e70 2024-03-25T19:18:10.144 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.144+0000 7f5bdb7fe640 1 -- 172.21.15.6:0/2173318909 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f5bec001d10 con 0x7f5bf40b4e70 2024-03-25T19:18:10.144 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.144+0000 7f5bdb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:18:10.144 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.144+0000 7f5bdb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:18:10.144 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.144+0000 7f5bdb7fe640 10 monclient(hunting): dump: 2024-03-25T19:18:10.144 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:18:10.144 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:10.144 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:10.144 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:10.144 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:18:10.144 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:10.145 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:18:10.145 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:18:10.145 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:18:10.145 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:10.145 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.144+0000 7f5bdb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:18:10.149 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.148+0000 7f5bdb7fe640 1 -- 172.21.15.6:0/2173318909 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 473464348 0 0) 0x7f5bf413cae0 con 0x7f5bf40b44b0 2024-03-25T19:18:10.149 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.149+0000 7f5bdb7fe640 1 -- 172.21.15.6:0/2173318909 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5be0004eb0 con 0x7f5bf40b44b0 2024-03-25T19:18:10.149 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.149+0000 7f5bdb7fe640 1 -- 172.21.15.6:0/2173318909 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f5be80018a0 con 0x7f5bf40b44b0 2024-03-25T19:18:10.149 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.149+0000 7f5bdb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:18:10.149 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.149+0000 7f5bdb7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:18:10.149 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.149+0000 7f5bdb7fe640 10 monclient(hunting): dump: 2024-03-25T19:18:10.149 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:18:10.149 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:10.149 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:10.149 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:10.149 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:18:10.149 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:10.149 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:18:10.150 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:18:10.150 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:18:10.150 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:10.161 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.161+0000 7f5bdb7fe640 1 -- 172.21.15.6:0/2173318909 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1987635461 0 0) 0x7f5bf41dad30 con 0x7f5bf40b1b00 2024-03-25T19:18:10.161 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.161+0000 7f5bdb7fe640 1 -- 172.21.15.6:0/2173318909 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5bf413cae0 con 0x7f5bf40b1b00 2024-03-25T19:18:10.161 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.161+0000 7f5bdb7fe640 1 -- 172.21.15.6:0/2173318909 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f5be4001b00 con 0x7f5bf40b1b00 2024-03-25T19:18:10.161 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.161+0000 7f5bdb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:18:10.161 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.161+0000 7f5bdb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:18:10.161 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.161+0000 7f5bdb7fe640 10 monclient(hunting): dump: 2024-03-25T19:18:10.161 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:18:10.162 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:10.162 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:10.162 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:10.162 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:18:10.162 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:10.162 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:18:10.162 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:18:10.162 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:18:10.162 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:10.166 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.165+0000 7f5bdb7fe640 1 -- 172.21.15.6:0/2173318909 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2753928635 0 0) 0x7f5bf413b940 con 0x7f5bf40b4e70 2024-03-25T19:18:10.166 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.165+0000 7f5bdb7fe640 1 -- 172.21.15.6:0/2173318909 >> v1:172.21.15.6:6790/0 conn(0x7f5bf40b44b0 legacy=0x7f5bf41db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:10.166 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.165+0000 7f5bdb7fe640 1 -- 172.21.15.6:0/2173318909 >> v1:172.21.15.6:6789/0 conn(0x7f5bf40b1b00 legacy=0x7f5bf413b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:10.166 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.165+0000 7f5bdb7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:18:10.166 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.165+0000 7f5bdb7fe640 1 monclient: found mon.b 2024-03-25T19:18:10.166 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.165+0000 7f5bdb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:18:10.166 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.165+0000 7f5bdb7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:18:10.166 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.165+0000 7f5bdb7fe640 1 -- 172.21.15.6:0/2173318909 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bf41db960 con 0x7f5bf40b4e70 2024-03-25T19:18:10.166 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.165+0000 7f5bdb7fe640 10 monclient: _finish_auth 0 2024-03-25T19:18:10.166 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.165+0000 7f5bdb7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:18:10.166 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.165+0000 7f5bdb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:18:10.166 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.165+0000 7f5bf904e640 5 monclient: authenticate success, global_id 4408 2024-03-25T19:18:10.167 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.165+0000 7f5bf904e640 10 monclient: _renew_subs 2024-03-25T19:18:10.167 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.165+0000 7f5bf904e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:18:10.167 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.165+0000 7f5bf904e640 1 -- 172.21.15.6:0/2173318909 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bf413a990 con 0x7f5bf40b4e70 2024-03-25T19:18:10.167 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.165+0000 7f5bf904e640 10 client.4408.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:18:10.167 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.165+0000 7f5bf904e640 10 monclient: _renew_subs 2024-03-25T19:18:10.167 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.165+0000 7f5bf904e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:18:10.167 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.165+0000 7f5bf904e640 1 -- 172.21.15.6:0/2173318909 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bf41e3720 con 0x7f5bf40b4e70 2024-03-25T19:18:10.167 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.165+0000 7f5bf904e640 1 librados: init done 2024-03-25T19:18:10.168 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.166+0000 7f5bf904e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:18:10.168 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.166+0000 7f5bf904e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:18:10.168 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.166+0000 7f5bf904e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:18:10.168 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.166+0000 7f5bf904e640 1 -- 172.21.15.6:0/2173318909 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bf4143660 con 0x7f5bf40b4e70 2024-03-25T19:18:10.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.442+0000 7f5bdb7fe640 1 -- 172.21.15.6:0/2173318909 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5bec003990 con 0x7f5bf40b4e70 2024-03-25T19:18:10.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.442+0000 7f5bdb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:18:10.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.442+0000 7f5bdb7fe640 1 -- 172.21.15.6:0/2173318909 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f5bec003d60 con 0x7f5bf40b4e70 2024-03-25T19:18:10.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.442+0000 7f5bdb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:18:10.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.442+0000 7f5bdb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:18:10.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.442+0000 7f5bdb7fe640 10 monclient: dump: 2024-03-25T19:18:10.443 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:18:10.443 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:10.443 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:10.443 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:10.444 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:18:10.444 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:10.444 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:18:10.444 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:18:10.444 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:18:10.444 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:10.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.442+0000 7f5bf159a640 4 set_mon_vals no callback set 2024-03-25T19:18:10.445 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.443+0000 7f5bdb7fe640 1 -- 172.21.15.6:0/2173318909 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f5bec003f80 con 0x7f5bf40b4e70 2024-03-25T19:18:10.445 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.444+0000 7f5bf2d9d640 10 monclient: get_auth_request con 0x7f5be006e0f0 auth_method 0 2024-03-25T19:18:10.445 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.444+0000 7f5bdb7fe640 10 client.4408.objecter ms_handle_connect 0x7f5be006e0f0 2024-03-25T19:18:10.447 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.447+0000 7f5bdb7fe640 1 -- 172.21.15.6:0/2173318909 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f5bec054b10 con 0x7f5bf40b4e70 2024-03-25T19:18:10.447 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.447+0000 7f5bdb7fe640 10 client.4408.objecter ms_dispatch 0x7f5bf4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:18:10.447 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.447+0000 7f5bdb7fe640 3 client.4408.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:18:10.447 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.447+0000 7f5bdb7fe640 3 client.4408.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:18:10.447 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.447+0000 7f5bdb7fe640 20 client.4408.objecter dump_active .. 0 homeless 2024-03-25T19:18:10.448 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.448+0000 7f5bdb7fe640 1 -- 172.21.15.6:0/2173318909 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5bec050620 con 0x7f5bf40b4e70 2024-03-25T19:18:10.448 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.448+0000 7f5bdb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:18:10.448 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.448+0000 7f5bdb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:18:10.698 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.698+0000 7f5bf904e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:18:10.698 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.698+0000 7f5bf904e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:18:10.698 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.698+0000 7f5bf904e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:18:10.698 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.698+0000 7f5bf904e640 1 -- 172.21.15.6:0/2173318909 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5bf404f730 con 0x7f5bf40b4e70 2024-03-25T19:18:10.735 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.735+0000 7f5bdb7fe640 1 -- 172.21.15.6:0/2173318909 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f5bec052550 con 0x7f5bf40b4e70 2024-03-25T19:18:10.736 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.735+0000 7f5bdb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:18:10.736 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.735+0000 7f5bdb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:18:10.736 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:18:10.736 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:18:10.740 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.738+0000 7f5bf904e640 10 librados: watch_flush enter 2024-03-25T19:18:10.740 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.738+0000 7f5bf904e640 10 librados: watch_flush exit 2024-03-25T19:18:10.740 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.738+0000 7f5bf904e640 20 client.4408.objecter shutdown clearing up homeless session... 2024-03-25T19:18:10.740 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.738+0000 7f5bf904e640 10 client.4408.objecter successfully canceled tick 2024-03-25T19:18:10.740 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.738+0000 7f5bf904e640 1 -- 172.21.15.6:0/2173318909 >> v1:172.21.15.6:6800/2936652650 conn(0x7f5be006e0f0 legacy=0x7f5be00705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:10.740 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.738+0000 7f5bf904e640 10 monclient: shutdown 2024-03-25T19:18:10.740 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.738+0000 7f5bf904e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:18:10.740 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.738+0000 7f5bf904e640 1 -- 172.21.15.6:0/2173318909 >> v1:172.21.15.192:6789/0 conn(0x7f5bf40b4e70 legacy=0x7f5bf41e2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:10.740 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.738+0000 7f5bf904e640 1 -- 172.21.15.6:0/2173318909 shutdown_connections 2024-03-25T19:18:10.740 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.738+0000 7f5bf904e640 1 -- 172.21.15.6:0/2173318909 >> 172.21.15.6:0/2173318909 conn(0x7f5bf40617a0 msgr2=0x7f5bf40b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:18:10.740 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.739+0000 7f5bf904e640 1 -- 172.21.15.6:0/2173318909 shutdown_connections 2024-03-25T19:18:10.740 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.739+0000 7f5bf904e640 1 -- 172.21.15.6:0/2173318909 wait complete. 2024-03-25T19:18:10.741 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:10.739+0000 7f5bf904e640 1 librados: shutdown 2024-03-25T19:18:10.750 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:18:17.556 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:18:17.672 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.672+0000 7f8acf910640 10 monclient: get_monmap_and_config 2024-03-25T19:18:17.672 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.672+0000 7f8acf910640 10 monclient: build_initial_monmap 2024-03-25T19:18:17.672 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.672+0000 7f8acf910640 10 monclient: monmap: 2024-03-25T19:18:17.672 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:18:17.672 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:17.672 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:18:17.673040+0000 2024-03-25T19:18:17.672 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:18:17.673040+0000 2024-03-25T19:18:17.672 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:18:17.672 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:17.672 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:18:17.672 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:18:17.672 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:18:17.673 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:17.673 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.672+0000 7f8acf910640 1 Processor -- start 2024-03-25T19:18:17.674 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.672+0000 7f8acf910640 1 -- start start 2024-03-25T19:18:17.674 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.672+0000 7f8acf910640 10 monclient: init 2024-03-25T19:18:17.674 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.673+0000 7f8acf910640 10 monclient: _reopen_session rank -1 2024-03-25T19:18:17.674 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.673+0000 7f8acf910640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:18:17.674 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.673+0000 7f8acf910640 10 monclient(hunting): picked mon.noname-c con 0x7f8ac80703b0 addr v1:172.21.15.192:6789/0 2024-03-25T19:18:17.674 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.673+0000 7f8acf910640 10 monclient(hunting): picked mon.noname-a con 0x7f8ac8140ba0 addr v1:172.21.15.6:6789/0 2024-03-25T19:18:17.674 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.673+0000 7f8acf910640 10 monclient(hunting): picked mon.noname-b con 0x7f8ac80af6e0 addr v1:172.21.15.6:6790/0 2024-03-25T19:18:17.674 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.673+0000 7f8acf910640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac80723b0 con 0x7f8ac8140ba0 2024-03-25T19:18:17.674 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.673+0000 7f8acf910640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac8141020 con 0x7f8ac80703b0 2024-03-25T19:18:17.674 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.673+0000 7f8acf910640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac8072600 con 0x7f8ac80af6e0 2024-03-25T19:18:17.674 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.673+0000 7f8acf910640 10 monclient(hunting): _renew_subs 2024-03-25T19:18:17.674 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.673+0000 7f8acf910640 10 monclient(hunting): authenticate will time out at 6201.913608s 2024-03-25T19:18:17.685 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.685+0000 7f8acd6ac640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f8ac80703b0 0x7f8ac8070790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:48220/0 (socket says 172.21.15.6:48220) 2024-03-25T19:18:17.685 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.685+0000 7f8acd6ac640 1 -- 172.21.15.6:0/2290198021 learned_addr learned my addr 172.21.15.6:0/2290198021 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:18:17.699 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.699+0000 7f8abffff640 1 -- 172.21.15.6:0/2290198021 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 813540022 0 0) 0x7f8ac80723b0 con 0x7f8ac8140ba0 2024-03-25T19:18:17.699 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.699+0000 7f8abffff640 10 monclient(hunting): _negotiate 2024-03-25T19:18:17.699 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.699+0000 7f8abffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:17.699 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.699+0000 7f8abffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:17.699 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.699+0000 7f8abffff640 10 monclient(hunting): my global_id is 4407 2024-03-25T19:18:17.699 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.699+0000 7f8abffff640 1 -- 172.21.15.6:0/2290198021 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ab4003520 con 0x7f8ac8140ba0 2024-03-25T19:18:17.704 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.704+0000 7f8abffff640 1 -- 172.21.15.6:0/2290198021 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 99982923 0 0) 0x7f8ac8141020 con 0x7f8ac80703b0 2024-03-25T19:18:17.704 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.704+0000 7f8abffff640 10 monclient(hunting): _negotiate 2024-03-25T19:18:17.704 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.704+0000 7f8abffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:17.704 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.704+0000 7f8abffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:17.704 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.704+0000 7f8abffff640 10 monclient(hunting): my global_id is 4411 2024-03-25T19:18:17.704 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.704+0000 7f8abffff640 1 -- 172.21.15.6:0/2290198021 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ac80723b0 con 0x7f8ac80703b0 2024-03-25T19:18:17.708 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.708+0000 7f8abffff640 1 -- 172.21.15.6:0/2290198021 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3613541416 0 0) 0x7f8ac8072600 con 0x7f8ac80af6e0 2024-03-25T19:18:17.708 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.708+0000 7f8abffff640 10 monclient(hunting): _negotiate 2024-03-25T19:18:17.708 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.708+0000 7f8abffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:17.708 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.708+0000 7f8abffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:17.708 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.708+0000 7f8abffff640 10 monclient(hunting): my global_id is 4409 2024-03-25T19:18:17.709 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.708+0000 7f8abffff640 1 -- 172.21.15.6:0/2290198021 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ac8141020 con 0x7f8ac80af6e0 2024-03-25T19:18:17.713 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.713+0000 7f8abffff640 1 -- 172.21.15.6:0/2290198021 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3004708305 0 0) 0x7f8ab4003520 con 0x7f8ac8140ba0 2024-03-25T19:18:17.713 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.713+0000 7f8abffff640 1 -- 172.21.15.6:0/2290198021 >> v1:172.21.15.6:6790/0 conn(0x7f8ac80af6e0 legacy=0x7f8ac814b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:17.713 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.713+0000 7f8abffff640 1 -- 172.21.15.6:0/2290198021 >> v1:172.21.15.192:6789/0 conn(0x7f8ac80703b0 legacy=0x7f8ac8070790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:17.713 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.713+0000 7f8abffff640 10 monclient: _finish_hunting 0 2024-03-25T19:18:17.713 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.713+0000 7f8abffff640 1 monclient: found mon.noname-a 2024-03-25T19:18:17.713 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.713+0000 7f8abffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:18:17.713 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.713+0000 7f8abffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-03-25T19:18:17.713 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.713+0000 7f8abffff640 1 -- 172.21.15.6:0/2290198021 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac80728e0 con 0x7f8ac8140ba0 2024-03-25T19:18:17.714 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.713+0000 7f8abffff640 10 monclient: _finish_auth 0 2024-03-25T19:18:17.714 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.713+0000 7f8abffff640 10 monclient: _check_auth_tickets 2024-03-25T19:18:17.714 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.713+0000 7f8abffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:18:17.714 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.713+0000 7f8abffff640 1 -- 172.21.15.6:0/2290198021 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f8ab8001690 con 0x7f8ac8140ba0 2024-03-25T19:18:17.714 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.713+0000 7f8abffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:18:17.714 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.713+0000 7f8abffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:18:17.714 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.713+0000 7f8abffff640 10 monclient: dump: 2024-03-25T19:18:17.714 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:18:17.714 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:17.714 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:17.715 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:17.715 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:18:17.715 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:17.715 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:18:17.715 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:18:17.715 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:18:17.715 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:17.715 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.713+0000 7f8acf910640 5 monclient: authenticate success, global_id 4407 2024-03-25T19:18:17.715 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.713+0000 7f8acf910640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:18:17.739 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.739+0000 7f8abffff640 1 -- 172.21.15.6:0/2290198021 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ab80018b0 con 0x7f8ac8140ba0 2024-03-25T19:18:17.740 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.739+0000 7f8abffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:18:17.740 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.739+0000 7f8abffff640 1 -- 172.21.15.6:0/2290198021 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f8ab8001bc0 con 0x7f8ac8140ba0 2024-03-25T19:18:17.740 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.739+0000 7f8acf910640 10 monclient: get_monmap_and_config success 2024-03-25T19:18:17.740 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.739+0000 7f8acf910640 4 set_mon_vals no callback set 2024-03-25T19:18:17.740 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.739+0000 7f8abffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:18:17.740 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.739+0000 7f8abffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:18:17.740 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.739+0000 7f8abffff640 10 monclient: dump: 2024-03-25T19:18:17.740 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:18:17.740 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:17.740 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:17.740 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:17.740 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:18:17.740 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:17.740 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:18:17.740 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:18:17.741 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:18:17.741 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:17.741 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.740+0000 7f8acf910640 10 monclient: shutdown 2024-03-25T19:18:17.741 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.740+0000 7f8acf910640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:18:17.741 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.740+0000 7f8acf910640 1 -- 172.21.15.6:0/2290198021 >> v1:172.21.15.6:6789/0 conn(0x7f8ac8140ba0 legacy=0x7f8ac8139cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:17.741 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.741+0000 7f8acf910640 1 -- 172.21.15.6:0/2290198021 shutdown_connections 2024-03-25T19:18:17.741 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.741+0000 7f8acf910640 1 -- 172.21.15.6:0/2290198021 >> 172.21.15.6:0/2290198021 conn(0x7f8ac80b3cd0 msgr2=0x7f8ac80b40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:18:17.742 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.741+0000 7f8acf910640 1 -- 172.21.15.6:0/2290198021 shutdown_connections 2024-03-25T19:18:17.742 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.741+0000 7f8acf910640 1 -- 172.21.15.6:0/2290198021 wait complete. 2024-03-25T19:18:17.742 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.742+0000 7f8acf910640 10 monclient: build_initial_monmap 2024-03-25T19:18:17.742 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.742+0000 7f8acf910640 10 monclient: monmap: 2024-03-25T19:18:17.743 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:18:17.743 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:18:17.743 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:18:17.743 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:18:17.743 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:18:17.743 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:18:17.743 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:18:17.743 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:18:17.743 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:18:17.743 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:17.743 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.742+0000 7f8acf910640 1 librados: starting msgr at 2024-03-25T19:18:17.743 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.742+0000 7f8acf910640 1 librados: starting objecter 2024-03-25T19:18:17.743 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.742+0000 7f8acf910640 1 Processor -- start 2024-03-25T19:18:17.743 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.742+0000 7f8acf910640 1 -- start start 2024-03-25T19:18:17.743 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.742+0000 7f8acf910640 1 librados: setting wanted keys 2024-03-25T19:18:17.743 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.742+0000 7f8acf910640 1 librados: calling monclient init 2024-03-25T19:18:17.744 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.742+0000 7f8acf910640 10 monclient: init 2024-03-25T19:18:17.744 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.743+0000 7f8acf910640 10 monclient: _reopen_session rank -1 2024-03-25T19:18:17.745 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.743+0000 7f8acf910640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:18:17.745 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.743+0000 7f8acf910640 10 monclient(hunting): picked mon.noname-a con 0x7f8ac80703b0 addr v1:172.21.15.6:6789/0 2024-03-25T19:18:17.745 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.743+0000 7f8acf910640 10 monclient(hunting): picked mon.noname-c con 0x7f8ac80af6e0 addr v1:172.21.15.6:6790/0 2024-03-25T19:18:17.745 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.743+0000 7f8acf910640 10 monclient(hunting): picked mon.noname-b con 0x7f8ac8140ba0 addr v1:172.21.15.192:6789/0 2024-03-25T19:18:17.745 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.743+0000 7f8acf910640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac81db930 con 0x7f8ac80703b0 2024-03-25T19:18:17.745 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.743+0000 7f8acf910640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac813d410 con 0x7f8ac8140ba0 2024-03-25T19:18:17.745 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.743+0000 7f8acf910640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac813d680 con 0x7f8ac80af6e0 2024-03-25T19:18:17.745 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.743+0000 7f8acf910640 10 monclient(hunting): _renew_subs 2024-03-25T19:18:17.745 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.743+0000 7f8acf910640 10 monclient(hunting): authenticate will time out at 6201.983525s 2024-03-25T19:18:17.752 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.752+0000 7f8acd6ac640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8ac80703b0 0x7f8ac813b570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58546/0 (socket says 172.21.15.6:58546) 2024-03-25T19:18:17.752 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.752+0000 7f8acd6ac640 1 -- 172.21.15.6:0/507823897 learned_addr learned my addr 172.21.15.6:0/507823897 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:18:17.754 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.753+0000 7f8abdffb640 10 client.?.objecter ms_handle_connect 0x7f8ac80703b0 2024-03-25T19:18:17.754 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.753+0000 7f8abdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:18:17.760 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.760+0000 7f8abdffb640 1 -- 172.21.15.6:0/507823897 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3422389217 0 0) 0x7f8ac81db930 con 0x7f8ac80703b0 2024-03-25T19:18:17.760 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.760+0000 7f8abdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:18:17.760 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.760+0000 7f8abdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:17.760 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.760+0000 7f8abdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:17.760 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.760+0000 7f8abdffb640 10 monclient(hunting): my global_id is 4410 2024-03-25T19:18:17.760 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.760+0000 7f8abdffb640 1 -- 172.21.15.6:0/507823897 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8aac004eb0 con 0x7f8ac80703b0 2024-03-25T19:18:17.761 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.761+0000 7f8abdffb640 10 client.?.objecter ms_handle_connect 0x7f8ac8140ba0 2024-03-25T19:18:17.761 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.761+0000 7f8abdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:18:17.763 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.763+0000 7f8abdffb640 10 client.?.objecter ms_handle_connect 0x7f8ac80af6e0 2024-03-25T19:18:17.763 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.763+0000 7f8abdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:18:17.770 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.770+0000 7f8abdffb640 1 -- 172.21.15.6:0/507823897 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3334376923 0 0) 0x7f8ac813d410 con 0x7f8ac8140ba0 2024-03-25T19:18:17.770 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.770+0000 7f8abdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:18:17.770 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.770+0000 7f8abdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:17.770 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.770+0000 7f8abdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:17.770 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.770+0000 7f8abdffb640 10 monclient(hunting): my global_id is 4414 2024-03-25T19:18:17.770 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.770+0000 7f8abdffb640 1 -- 172.21.15.6:0/507823897 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ac81db930 con 0x7f8ac8140ba0 2024-03-25T19:18:17.773 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.773+0000 7f8abdffb640 1 -- 172.21.15.6:0/507823897 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2379472418 0 0) 0x7f8ac813d680 con 0x7f8ac80af6e0 2024-03-25T19:18:17.773 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.773+0000 7f8abdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:18:17.773 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.773+0000 7f8abdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:17.773 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.773+0000 7f8abdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:17.773 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.773+0000 7f8abdffb640 10 monclient(hunting): my global_id is 4412 2024-03-25T19:18:17.773 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.773+0000 7f8abdffb640 1 -- 172.21.15.6:0/507823897 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ac813d410 con 0x7f8ac80af6e0 2024-03-25T19:18:17.776 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.776+0000 7f8abdffb640 1 -- 172.21.15.6:0/507823897 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 506607155 0 0) 0x7f8aac004eb0 con 0x7f8ac80703b0 2024-03-25T19:18:17.777 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.776+0000 7f8abdffb640 1 -- 172.21.15.6:0/507823897 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ac813d680 con 0x7f8ac80703b0 2024-03-25T19:18:17.777 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.776+0000 7f8abdffb640 1 -- 172.21.15.6:0/507823897 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f8ac4001970 con 0x7f8ac80703b0 2024-03-25T19:18:17.777 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.776+0000 7f8abdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:18:17.777 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.776+0000 7f8abdffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:18:17.777 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.776+0000 7f8abdffb640 10 monclient(hunting): dump: 2024-03-25T19:18:17.777 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:18:17.777 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:17.777 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:17.777 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:17.777 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:18:17.777 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:17.777 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:18:17.777 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:18:17.777 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:18:17.778 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:17.778 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.776+0000 7f8abdffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:18:17.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.782+0000 7f8abdffb640 1 -- 172.21.15.6:0/507823897 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2994606999 0 0) 0x7f8ac813d680 con 0x7f8ac80703b0 2024-03-25T19:18:17.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.782+0000 7f8abdffb640 1 -- 172.21.15.6:0/507823897 >> v1:172.21.15.6:6790/0 conn(0x7f8ac80af6e0 legacy=0x7f8ac81dbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:17.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.782+0000 7f8abdffb640 1 -- 172.21.15.6:0/507823897 >> v1:172.21.15.192:6789/0 conn(0x7f8ac8140ba0 legacy=0x7f8ac80aca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:17.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.782+0000 7f8abdffb640 10 monclient: _finish_hunting 0 2024-03-25T19:18:17.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.782+0000 7f8abdffb640 1 monclient: found mon.a 2024-03-25T19:18:17.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.782+0000 7f8abdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:18:17.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.782+0000 7f8abdffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:18:17.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.782+0000 7f8abdffb640 1 -- 172.21.15.6:0/507823897 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac806de70 con 0x7f8ac80703b0 2024-03-25T19:18:17.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.782+0000 7f8abdffb640 10 monclient: _finish_auth 0 2024-03-25T19:18:17.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.782+0000 7f8abdffb640 10 monclient: _check_auth_tickets 2024-03-25T19:18:17.784 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.782+0000 7f8abdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:18:17.784 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.782+0000 7f8acf910640 5 monclient: authenticate success, global_id 4410 2024-03-25T19:18:17.784 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.782+0000 7f8acf910640 10 monclient: _renew_subs 2024-03-25T19:18:17.784 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.782+0000 7f8acf910640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:18:17.784 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.782+0000 7f8acf910640 1 -- 172.21.15.6:0/507823897 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ac81dc4d0 con 0x7f8ac80703b0 2024-03-25T19:18:17.784 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.783+0000 7f8acf910640 10 client.4410.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:18:17.784 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.783+0000 7f8acf910640 10 monclient: _renew_subs 2024-03-25T19:18:17.784 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.783+0000 7f8acf910640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:18:17.784 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.783+0000 7f8acf910640 1 -- 172.21.15.6:0/507823897 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ac813c220 con 0x7f8ac80703b0 2024-03-25T19:18:17.784 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.783+0000 7f8acf910640 1 librados: init done 2024-03-25T19:18:17.785 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.784+0000 7f8acf910640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:18:17.785 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.784+0000 7f8acf910640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:18:17.785 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.784+0000 7f8acf910640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:18:17.785 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:17.784+0000 7f8acf910640 1 -- 172.21.15.6:0/507823897 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ac813e500 con 0x7f8ac80703b0 2024-03-25T19:18:18.143 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:18.143+0000 7f8abdffb640 1 -- 172.21.15.6:0/507823897 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ac40047e0 con 0x7f8ac80703b0 2024-03-25T19:18:18.143 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:18.143+0000 7f8abdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:18:18.144 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:18.143+0000 7f8abdffb640 1 -- 172.21.15.6:0/507823897 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f8ac4004af0 con 0x7f8ac80703b0 2024-03-25T19:18:18.144 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:18.143+0000 7f8abdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:18:18.144 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:18.143+0000 7f8abdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:18:18.144 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:18.143+0000 7f8abdffb640 10 monclient: dump: 2024-03-25T19:18:18.144 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:18:18.144 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:18.144 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:18.144 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:18.144 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:18:18.144 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:18.144 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:18:18.144 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:18:18.144 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:18:18.144 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:18.145 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:18.143+0000 7f8abf7fe640 4 set_mon_vals no callback set 2024-03-25T19:18:18.145 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:18.144+0000 7f8abdffb640 1 -- 172.21.15.6:0/507823897 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f8ac401ab00 con 0x7f8ac80703b0 2024-03-25T19:18:18.145 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:18.145+0000 7f8acceab640 10 monclient: get_auth_request con 0x7f8aac06ddf0 auth_method 0 2024-03-25T19:18:18.145 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:18.145+0000 7f8abdffb640 10 client.4410.objecter ms_handle_connect 0x7f8aac06ddf0 2024-03-25T19:18:18.150 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:18.150+0000 7f8abdffb640 1 -- 172.21.15.6:0/507823897 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f8ac40547e0 con 0x7f8ac80703b0 2024-03-25T19:18:18.151 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:18.150+0000 7f8abdffb640 10 client.4410.objecter ms_dispatch 0x7f8ac8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:18:18.151 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:18.150+0000 7f8abdffb640 3 client.4410.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:18:18.151 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:18.150+0000 7f8abdffb640 3 client.4410.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:18:18.151 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:18.151+0000 7f8abdffb640 20 client.4410.objecter dump_active .. 0 homeless 2024-03-25T19:18:18.151 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:18.151+0000 7f8abdffb640 1 -- 172.21.15.6:0/507823897 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8ac4050120 con 0x7f8ac80703b0 2024-03-25T19:18:18.151 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:18.151+0000 7f8abdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:18:18.151 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:18.151+0000 7f8abdffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:18:18.377 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:18.377+0000 7f8acf910640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:18:18.377 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:18.377+0000 7f8acf910640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:18:18.377 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:18.377+0000 7f8acf910640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:18:18.377 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:18.377+0000 7f8acf910640 1 -- 172.21.15.6:0/507823897 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ac8140380 con 0x7f8ac80703b0 2024-03-25T19:18:18.466 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:18.466+0000 7f8abdffb640 1 -- 172.21.15.6:0/507823897 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f8ac4052220 con 0x7f8ac80703b0 2024-03-25T19:18:18.466 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:18.466+0000 7f8abdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:18:18.466 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:18.466+0000 7f8abdffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:18:18.467 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:18:18.467 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:18:18.469 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:18.468+0000 7f8acf910640 10 librados: watch_flush enter 2024-03-25T19:18:18.469 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:18.468+0000 7f8acf910640 10 librados: watch_flush exit 2024-03-25T19:18:18.469 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:18.468+0000 7f8acf910640 20 client.4410.objecter shutdown clearing up homeless session... 2024-03-25T19:18:18.469 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:18.468+0000 7f8acf910640 10 client.4410.objecter successfully canceled tick 2024-03-25T19:18:18.469 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:18.468+0000 7f8acf910640 1 -- 172.21.15.6:0/507823897 >> v1:172.21.15.6:6800/2936652650 conn(0x7f8aac06ddf0 legacy=0x7f8aac0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:18.469 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:18.468+0000 7f8acf910640 10 monclient: shutdown 2024-03-25T19:18:18.469 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:18.468+0000 7f8acf910640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:18:18.469 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:18.468+0000 7f8acf910640 1 -- 172.21.15.6:0/507823897 >> v1:172.21.15.6:6789/0 conn(0x7f8ac80703b0 legacy=0x7f8ac813b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:18.469 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:18.468+0000 7f8acf910640 1 -- 172.21.15.6:0/507823897 shutdown_connections 2024-03-25T19:18:18.469 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:18.468+0000 7f8acf910640 1 -- 172.21.15.6:0/507823897 >> 172.21.15.6:0/507823897 conn(0x7f8ac80b3cd0 msgr2=0x7f8ac8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:18:18.469 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:18.469+0000 7f8acf910640 1 -- 172.21.15.6:0/507823897 shutdown_connections 2024-03-25T19:18:18.469 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:18.469+0000 7f8acf910640 1 -- 172.21.15.6:0/507823897 wait complete. 2024-03-25T19:18:18.469 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:18.469+0000 7f8acf910640 1 librados: shutdown 2024-03-25T19:18:18.480 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:18:23.223 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:04:40.569 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:18:23.225 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:04:40.572 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:18:23.273 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:04:40.620 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:18:23.279 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:04:40.597 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:18:23.285 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:04:40.603 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:18:23.289 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:04:40.636 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:18:23.296 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:04:40.643 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:18:23.318 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:04:40.636 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:18:23.319 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:04:40.666 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:18:25.287 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:18:25.402 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.402+0000 7ff460663640 10 monclient: get_monmap_and_config 2024-03-25T19:18:25.402 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.402+0000 7ff460663640 10 monclient: build_initial_monmap 2024-03-25T19:18:25.402 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.402+0000 7ff460663640 10 monclient: monmap: 2024-03-25T19:18:25.402 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:18:25.402 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:25.403 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:18:25.403215+0000 2024-03-25T19:18:25.403 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:18:25.403215+0000 2024-03-25T19:18:25.403 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:18:25.403 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:25.403 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:18:25.403 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:18:25.403 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:18:25.403 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:25.403 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.403+0000 7ff460663640 1 Processor -- start 2024-03-25T19:18:25.404 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.403+0000 7ff460663640 1 -- start start 2024-03-25T19:18:25.404 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.403+0000 7ff460663640 10 monclient: init 2024-03-25T19:18:25.404 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.404+0000 7ff460663640 10 monclient: _reopen_session rank -1 2024-03-25T19:18:25.405 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.404+0000 7ff460663640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:18:25.405 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.404+0000 7ff460663640 10 monclient(hunting): picked mon.noname-b con 0x7ff4580b4ff0 addr v1:172.21.15.6:6790/0 2024-03-25T19:18:25.405 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.404+0000 7ff460663640 10 monclient(hunting): picked mon.noname-a con 0x7ff4580b4670 addr v1:172.21.15.6:6789/0 2024-03-25T19:18:25.405 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.404+0000 7ff460663640 10 monclient(hunting): picked mon.noname-c con 0x7ff45813c480 addr v1:172.21.15.192:6789/0 2024-03-25T19:18:25.405 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.404+0000 7ff460663640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4580ab0e0 con 0x7ff4580b4670 2024-03-25T19:18:25.405 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.404+0000 7ff460663640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4580ab3c0 con 0x7ff45813c480 2024-03-25T19:18:25.405 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.404+0000 7ff460663640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4580a9f30 con 0x7ff4580b4ff0 2024-03-25T19:18:25.405 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.404+0000 7ff460663640 10 monclient(hunting): _renew_subs 2024-03-25T19:18:25.405 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.404+0000 7ff460663640 10 monclient(hunting): authenticate will time out at 6209.644493s 2024-03-25T19:18:25.411 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.411+0000 7ff45dbfe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff4580b4670 0x7ff4581425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37764/0 (socket says 172.21.15.6:37764) 2024-03-25T19:18:25.411 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.411+0000 7ff45dbfe640 1 -- 172.21.15.6:0/363911214 learned_addr learned my addr 172.21.15.6:0/363911214 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:18:25.432 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.432+0000 7ff45d3fd640 1 -- 172.21.15.6:0/363911214 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2144933946 0 0) 0x7ff4580ab3c0 con 0x7ff45813c480 2024-03-25T19:18:25.433 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.432+0000 7ff45d3fd640 10 monclient(hunting): _negotiate 2024-03-25T19:18:25.433 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.432+0000 7ff45d3fd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:25.433 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.432+0000 7ff45d3fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:25.433 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.432+0000 7ff45d3fd640 10 monclient(hunting): my global_id is 4417 2024-03-25T19:18:25.433 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.432+0000 7ff45d3fd640 1 -- 172.21.15.6:0/363911214 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff444003520 con 0x7ff45813c480 2024-03-25T19:18:25.440 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.440+0000 7ff45d3fd640 1 -- 172.21.15.6:0/363911214 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 463337055 0 0) 0x7ff4580ab0e0 con 0x7ff4580b4670 2024-03-25T19:18:25.440 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.440+0000 7ff45d3fd640 10 monclient(hunting): _negotiate 2024-03-25T19:18:25.441 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.440+0000 7ff45d3fd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:25.441 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.440+0000 7ff45d3fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:25.441 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.440+0000 7ff45d3fd640 10 monclient(hunting): my global_id is 4413 2024-03-25T19:18:25.441 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.440+0000 7ff45d3fd640 1 -- 172.21.15.6:0/363911214 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4580ab3c0 con 0x7ff4580b4670 2024-03-25T19:18:25.453 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.452+0000 7ff45d3fd640 1 -- 172.21.15.6:0/363911214 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2699600056 0 0) 0x7ff444003520 con 0x7ff45813c480 2024-03-25T19:18:25.453 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.452+0000 7ff45d3fd640 1 -- 172.21.15.6:0/363911214 >> v1:172.21.15.6:6790/0 conn(0x7ff4580b4ff0 legacy=0x7ff45813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:25.453 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.453+0000 7ff45d3fd640 1 -- 172.21.15.6:0/363911214 >> v1:172.21.15.6:6789/0 conn(0x7ff4580b4670 legacy=0x7ff4581425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:25.453 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.453+0000 7ff45d3fd640 10 monclient: _finish_hunting 0 2024-03-25T19:18:25.453 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.453+0000 7ff45d3fd640 1 monclient: found mon.noname-c 2024-03-25T19:18:25.453 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.453+0000 7ff45d3fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:18:25.453 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.453+0000 7ff45d3fd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:18:25.453 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.453+0000 7ff45d3fd640 1 -- 172.21.15.6:0/363911214 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4580a9b70 con 0x7ff45813c480 2024-03-25T19:18:25.453 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.453+0000 7ff45d3fd640 10 monclient: _finish_auth 0 2024-03-25T19:18:25.453 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.453+0000 7ff45d3fd640 10 monclient: _check_auth_tickets 2024-03-25T19:18:25.453 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.453+0000 7ff45d3fd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:18:25.453 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.453+0000 7ff45d3fd640 1 -- 172.21.15.6:0/363911214 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7ff450001a10 con 0x7ff45813c480 2024-03-25T19:18:25.454 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.453+0000 7ff460663640 5 monclient: authenticate success, global_id 4417 2024-03-25T19:18:25.454 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.453+0000 7ff460663640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:18:25.454 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.453+0000 7ff45d3fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:18:25.454 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.453+0000 7ff45d3fd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:18:25.454 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.453+0000 7ff45d3fd640 10 monclient: dump: 2024-03-25T19:18:25.454 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:18:25.454 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:25.454 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:25.454 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:25.454 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:18:25.454 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:25.454 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:18:25.454 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:18:25.454 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:18:25.454 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:25.455 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.453+0000 7ff460663640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:18:25.464 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.464+0000 7ff45d3fd640 1 -- 172.21.15.6:0/363911214 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff450001b70 con 0x7ff45813c480 2024-03-25T19:18:25.464 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.464+0000 7ff45d3fd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:18:25.464 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.464+0000 7ff45d3fd640 1 -- 172.21.15.6:0/363911214 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7ff450001f80 con 0x7ff45813c480 2024-03-25T19:18:25.464 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.464+0000 7ff45d3fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:18:25.464 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.464+0000 7ff45d3fd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:18:25.464 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.464+0000 7ff45d3fd640 10 monclient: dump: 2024-03-25T19:18:25.464 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:18:25.464 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:25.464 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:25.465 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:25.465 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:18:25.465 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:25.465 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:18:25.465 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:18:25.465 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:18:25.465 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:25.465 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.464+0000 7ff460663640 10 monclient: get_monmap_and_config success 2024-03-25T19:18:25.465 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.464+0000 7ff460663640 4 set_mon_vals no callback set 2024-03-25T19:18:25.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.464+0000 7ff460663640 10 monclient: shutdown 2024-03-25T19:18:25.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.464+0000 7ff460663640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:18:25.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.464+0000 7ff460663640 1 -- 172.21.15.6:0/363911214 >> v1:172.21.15.192:6789/0 conn(0x7ff45813c480 legacy=0x7ff45814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:25.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.465+0000 7ff460663640 1 -- 172.21.15.6:0/363911214 shutdown_connections 2024-03-25T19:18:25.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.465+0000 7ff460663640 1 -- 172.21.15.6:0/363911214 >> 172.21.15.6:0/363911214 conn(0x7ff4580617a0 msgr2=0x7ff458061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:18:25.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.465+0000 7ff460663640 1 -- 172.21.15.6:0/363911214 shutdown_connections 2024-03-25T19:18:25.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.465+0000 7ff460663640 1 -- 172.21.15.6:0/363911214 wait complete. 2024-03-25T19:18:25.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.465+0000 7ff460663640 10 monclient: build_initial_monmap 2024-03-25T19:18:25.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.465+0000 7ff460663640 10 monclient: monmap: 2024-03-25T19:18:25.467 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:18:25.467 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:18:25.467 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:18:25.467 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:18:25.467 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:18:25.468 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:18:25.468 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:18:25.468 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:18:25.468 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:18:25.468 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:25.468 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.466+0000 7ff460663640 1 librados: starting msgr at 2024-03-25T19:18:25.468 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.466+0000 7ff460663640 1 librados: starting objecter 2024-03-25T19:18:25.468 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.466+0000 7ff460663640 1 Processor -- start 2024-03-25T19:18:25.468 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.466+0000 7ff460663640 1 -- start start 2024-03-25T19:18:25.468 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.466+0000 7ff460663640 1 librados: setting wanted keys 2024-03-25T19:18:25.468 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.466+0000 7ff460663640 1 librados: calling monclient init 2024-03-25T19:18:25.468 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.466+0000 7ff460663640 10 monclient: init 2024-03-25T19:18:25.468 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.466+0000 7ff460663640 10 monclient: _reopen_session rank -1 2024-03-25T19:18:25.468 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.466+0000 7ff460663640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:18:25.468 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.466+0000 7ff460663640 10 monclient(hunting): picked mon.noname-a con 0x7ff4580b4670 addr v1:172.21.15.6:6789/0 2024-03-25T19:18:25.468 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.466+0000 7ff460663640 10 monclient(hunting): picked mon.noname-b con 0x7ff4580b4ff0 addr v1:172.21.15.192:6789/0 2024-03-25T19:18:25.469 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.466+0000 7ff460663640 10 monclient(hunting): picked mon.noname-c con 0x7ff45813c480 addr v1:172.21.15.6:6790/0 2024-03-25T19:18:25.469 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.466+0000 7ff460663640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4580b0830 con 0x7ff4580b4670 2024-03-25T19:18:25.469 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.466+0000 7ff460663640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4580b0b10 con 0x7ff4580b4ff0 2024-03-25T19:18:25.469 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.466+0000 7ff460663640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4580b1550 con 0x7ff45813c480 2024-03-25T19:18:25.469 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.466+0000 7ff460663640 10 monclient(hunting): _renew_subs 2024-03-25T19:18:25.469 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.466+0000 7ff460663640 10 monclient(hunting): authenticate will time out at 6209.706665s 2024-03-25T19:18:25.480 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.480+0000 7ff45dbfe640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7ff4580b4ff0 0x7ff4580aef50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:54496/0 (socket says 172.21.15.6:54496) 2024-03-25T19:18:25.480 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.480+0000 7ff45dbfe640 1 -- 172.21.15.6:0/2849343753 learned_addr learned my addr 172.21.15.6:0/2849343753 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:18:25.482 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.482+0000 7ff44effd640 10 client.?.objecter ms_handle_connect 0x7ff4580b4ff0 2024-03-25T19:18:25.482 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.482+0000 7ff44effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:18:25.482 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.482+0000 7ff44effd640 10 client.?.objecter ms_handle_connect 0x7ff45813c480 2024-03-25T19:18:25.482 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.482+0000 7ff44effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:18:25.489 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.489+0000 7ff44effd640 10 client.?.objecter ms_handle_connect 0x7ff4580b4670 2024-03-25T19:18:25.489 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.489+0000 7ff44effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:18:25.492 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.492+0000 7ff44effd640 1 -- 172.21.15.6:0/2849343753 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4256910354 0 0) 0x7ff4580b0b10 con 0x7ff4580b4ff0 2024-03-25T19:18:25.492 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.492+0000 7ff44effd640 10 monclient(hunting): _negotiate 2024-03-25T19:18:25.492 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.492+0000 7ff44effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:25.492 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.492+0000 7ff44effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:25.493 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.492+0000 7ff44effd640 10 monclient(hunting): my global_id is 4420 2024-03-25T19:18:25.493 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.492+0000 7ff44effd640 1 -- 172.21.15.6:0/2849343753 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff43c004eb0 con 0x7ff4580b4ff0 2024-03-25T19:18:25.500 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.500+0000 7ff44effd640 1 -- 172.21.15.6:0/2849343753 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2409870204 0 0) 0x7ff4580b1550 con 0x7ff45813c480 2024-03-25T19:18:25.500 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.500+0000 7ff44effd640 10 monclient(hunting): _negotiate 2024-03-25T19:18:25.500 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.500+0000 7ff44effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:25.500 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.500+0000 7ff44effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:25.500 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.500+0000 7ff44effd640 10 monclient(hunting): my global_id is 4418 2024-03-25T19:18:25.500 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.500+0000 7ff44effd640 1 -- 172.21.15.6:0/2849343753 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4580b0b10 con 0x7ff45813c480 2024-03-25T19:18:25.504 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.504+0000 7ff44effd640 1 -- 172.21.15.6:0/2849343753 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1269572308 0 0) 0x7ff43c004eb0 con 0x7ff4580b4ff0 2024-03-25T19:18:25.505 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.504+0000 7ff44effd640 1 -- 172.21.15.6:0/2849343753 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4580b1550 con 0x7ff4580b4ff0 2024-03-25T19:18:25.505 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.504+0000 7ff44effd640 1 -- 172.21.15.6:0/2849343753 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7ff4480019b0 con 0x7ff4580b4ff0 2024-03-25T19:18:25.505 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.504+0000 7ff44effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:18:25.505 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.504+0000 7ff44effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:18:25.505 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.504+0000 7ff44effd640 10 monclient(hunting): dump: 2024-03-25T19:18:25.505 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:18:25.505 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:25.505 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:25.505 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:25.505 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:18:25.505 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:25.505 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:18:25.505 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:18:25.505 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:18:25.505 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:25.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.504+0000 7ff44effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:18:25.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.505+0000 7ff44effd640 1 -- 172.21.15.6:0/2849343753 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2948613360 0 0) 0x7ff4580b0830 con 0x7ff4580b4670 2024-03-25T19:18:25.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.505+0000 7ff44effd640 10 monclient(hunting): _negotiate 2024-03-25T19:18:25.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.505+0000 7ff44effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:25.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.505+0000 7ff44effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:25.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.505+0000 7ff44effd640 10 monclient(hunting): my global_id is 4416 2024-03-25T19:18:25.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.505+0000 7ff44effd640 1 -- 172.21.15.6:0/2849343753 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff43c004eb0 con 0x7ff4580b4670 2024-03-25T19:18:25.528 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.528+0000 7ff44effd640 1 -- 172.21.15.6:0/2849343753 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2057751429 0 0) 0x7ff4580b1550 con 0x7ff4580b4ff0 2024-03-25T19:18:25.528 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.528+0000 7ff44effd640 1 -- 172.21.15.6:0/2849343753 >> v1:172.21.15.6:6790/0 conn(0x7ff45813c480 legacy=0x7ff4581e2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:25.528 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.528+0000 7ff44effd640 1 -- 172.21.15.6:0/2849343753 >> v1:172.21.15.6:6789/0 conn(0x7ff4580b4670 legacy=0x7ff4580ae720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:25.528 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.528+0000 7ff44effd640 10 monclient: _finish_hunting 0 2024-03-25T19:18:25.528 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.528+0000 7ff44effd640 1 monclient: found mon.b 2024-03-25T19:18:25.528 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.528+0000 7ff44effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:18:25.528 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.528+0000 7ff44effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:18:25.529 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.528+0000 7ff44effd640 1 -- 172.21.15.6:0/2849343753 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4580b0f30 con 0x7ff4580b4ff0 2024-03-25T19:18:25.529 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.528+0000 7ff44effd640 10 monclient: _finish_auth 0 2024-03-25T19:18:25.529 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.528+0000 7ff44effd640 10 monclient: _check_auth_tickets 2024-03-25T19:18:25.529 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.528+0000 7ff44effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:18:25.529 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.528+0000 7ff460663640 5 monclient: authenticate success, global_id 4420 2024-03-25T19:18:25.529 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.528+0000 7ff460663640 10 monclient: _renew_subs 2024-03-25T19:18:25.529 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.528+0000 7ff460663640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:18:25.529 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.528+0000 7ff460663640 1 -- 172.21.15.6:0/2849343753 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4580b1180 con 0x7ff4580b4ff0 2024-03-25T19:18:25.529 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.528+0000 7ff460663640 10 client.4420.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:18:25.529 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.528+0000 7ff460663640 10 monclient: _renew_subs 2024-03-25T19:18:25.529 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.528+0000 7ff460663640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:18:25.529 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.528+0000 7ff460663640 1 -- 172.21.15.6:0/2849343753 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4580af840 con 0x7ff4580b4ff0 2024-03-25T19:18:25.529 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.528+0000 7ff460663640 1 librados: init done 2024-03-25T19:18:25.530 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.529+0000 7ff460663640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:18:25.530 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.529+0000 7ff460663640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:18:25.530 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.529+0000 7ff460663640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:18:25.530 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.529+0000 7ff460663640 1 -- 172.21.15.6:0/2849343753 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4580b16e0 con 0x7ff4580b4ff0 2024-03-25T19:18:25.804 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.804+0000 7ff44effd640 1 -- 172.21.15.6:0/2849343753 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4480047e0 con 0x7ff4580b4ff0 2024-03-25T19:18:25.805 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.804+0000 7ff44effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:18:25.805 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.804+0000 7ff44effd640 1 -- 172.21.15.6:0/2849343753 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7ff448004b30 con 0x7ff4580b4ff0 2024-03-25T19:18:25.805 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.804+0000 7ff44effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:18:25.805 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.804+0000 7ff44effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:18:25.805 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.804+0000 7ff45cbfc640 4 set_mon_vals no callback set 2024-03-25T19:18:25.805 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.804+0000 7ff44effd640 10 monclient: dump: 2024-03-25T19:18:25.805 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:18:25.805 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:25.805 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:25.805 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:25.805 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:18:25.805 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:25.805 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:18:25.805 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:18:25.805 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:18:25.806 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:25.806 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.805+0000 7ff44effd640 1 -- 172.21.15.6:0/2849343753 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7ff448004d50 con 0x7ff4580b4ff0 2024-03-25T19:18:25.806 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.805+0000 7ff45e3ff640 10 monclient: get_auth_request con 0x7ff43c06de00 auth_method 0 2024-03-25T19:18:25.806 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.805+0000 7ff44effd640 10 client.4420.objecter ms_handle_connect 0x7ff43c06de00 2024-03-25T19:18:25.807 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.807+0000 7ff44effd640 1 -- 172.21.15.6:0/2849343753 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7ff448054890 con 0x7ff4580b4ff0 2024-03-25T19:18:25.807 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.807+0000 7ff44effd640 10 client.4420.objecter ms_dispatch 0x7ff458000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:18:25.808 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.807+0000 7ff44effd640 3 client.4420.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:18:25.808 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.807+0000 7ff44effd640 3 client.4420.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:18:25.808 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.807+0000 7ff44effd640 20 client.4420.objecter dump_active .. 0 homeless 2024-03-25T19:18:25.809 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.809+0000 7ff44effd640 1 -- 172.21.15.6:0/2849343753 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff4480503a0 con 0x7ff4580b4ff0 2024-03-25T19:18:25.809 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.809+0000 7ff44effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:18:25.809 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:25.809+0000 7ff44effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:18:26.043 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:26.043+0000 7ff460663640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:18:26.043 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:26.043+0000 7ff460663640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:18:26.043 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:26.043+0000 7ff460663640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:18:26.043 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:26.043+0000 7ff460663640 1 -- 172.21.15.6:0/2849343753 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff45806dbd0 con 0x7ff4580b4ff0 2024-03-25T19:18:26.066 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:26.066+0000 7ff44effd640 1 -- 172.21.15.6:0/2849343753 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7ff4480522d0 con 0x7ff4580b4ff0 2024-03-25T19:18:26.066 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:26.066+0000 7ff44effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:18:26.067 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:26.066+0000 7ff44effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:18:26.067 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:18:26.067 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:18:26.069 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:26.068+0000 7ff460663640 10 librados: watch_flush enter 2024-03-25T19:18:26.069 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:26.068+0000 7ff460663640 10 librados: watch_flush exit 2024-03-25T19:18:26.069 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:26.068+0000 7ff460663640 20 client.4420.objecter shutdown clearing up homeless session... 2024-03-25T19:18:26.070 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:26.068+0000 7ff460663640 10 client.4420.objecter successfully canceled tick 2024-03-25T19:18:26.070 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:26.069+0000 7ff460663640 1 -- 172.21.15.6:0/2849343753 >> v1:172.21.15.6:6800/2936652650 conn(0x7ff43c06de00 legacy=0x7ff43c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:26.070 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:26.069+0000 7ff460663640 10 monclient: shutdown 2024-03-25T19:18:26.070 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:26.069+0000 7ff460663640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:18:26.070 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:26.069+0000 7ff460663640 1 -- 172.21.15.6:0/2849343753 >> v1:172.21.15.192:6789/0 conn(0x7ff4580b4ff0 legacy=0x7ff4580aef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:26.070 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:26.069+0000 7ff460663640 1 -- 172.21.15.6:0/2849343753 shutdown_connections 2024-03-25T19:18:26.070 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:26.069+0000 7ff460663640 1 -- 172.21.15.6:0/2849343753 >> 172.21.15.6:0/2849343753 conn(0x7ff4580617a0 msgr2=0x7ff458145670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:18:26.070 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:26.069+0000 7ff460663640 1 -- 172.21.15.6:0/2849343753 shutdown_connections 2024-03-25T19:18:26.070 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:26.069+0000 7ff460663640 1 -- 172.21.15.6:0/2849343753 wait complete. 2024-03-25T19:18:26.070 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:26.069+0000 7ff460663640 1 librados: shutdown 2024-03-25T19:18:26.080 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:18:32.886 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:18:33.003 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.003+0000 7f1474ef6640 10 monclient: get_monmap_and_config 2024-03-25T19:18:33.003 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.003+0000 7f1474ef6640 10 monclient: build_initial_monmap 2024-03-25T19:18:33.003 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.003+0000 7f1474ef6640 10 monclient: monmap: 2024-03-25T19:18:33.003 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:18:33.003 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:33.004 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:18:33.004354+0000 2024-03-25T19:18:33.004 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:18:33.004354+0000 2024-03-25T19:18:33.004 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:18:33.004 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:33.004 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:18:33.004 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:18:33.004 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:18:33.004 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:33.005 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.004+0000 7f1474ef6640 1 Processor -- start 2024-03-25T19:18:33.005 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.004+0000 7f1474ef6640 1 -- start start 2024-03-25T19:18:33.005 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.004+0000 7f1474ef6640 10 monclient: init 2024-03-25T19:18:33.005 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.005+0000 7f1474ef6640 10 monclient: _reopen_session rank -1 2024-03-25T19:18:33.005 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.005+0000 7f1474ef6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:18:33.005 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.005+0000 7f1474ef6640 10 monclient(hunting): picked mon.noname-a con 0x7f14700b4e70 addr v1:172.21.15.6:6789/0 2024-03-25T19:18:33.005 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.005+0000 7f1474ef6640 10 monclient(hunting): picked mon.noname-c con 0x7f14700b44b0 addr v1:172.21.15.192:6789/0 2024-03-25T19:18:33.005 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.005+0000 7f1474ef6640 10 monclient(hunting): picked mon.noname-b con 0x7f14700b1b00 addr v1:172.21.15.6:6790/0 2024-03-25T19:18:33.005 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.005+0000 7f1474ef6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14700a96f0 con 0x7f14700b4e70 2024-03-25T19:18:33.006 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.005+0000 7f1474ef6640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14700ab0e0 con 0x7f14700b44b0 2024-03-25T19:18:33.006 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.005+0000 7f1474ef6640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1470145f70 con 0x7f14700b1b00 2024-03-25T19:18:33.006 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.005+0000 7f1474ef6640 10 monclient(hunting): _renew_subs 2024-03-25T19:18:33.006 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.005+0000 7f1474ef6640 10 monclient(hunting): authenticate will time out at 6217.245057s 2024-03-25T19:18:33.010 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.011+0000 7f146dd9b640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f14700b44b0 0x7f1470145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:39454/0 (socket says 172.21.15.6:39454) 2024-03-25T19:18:33.011 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.011+0000 7f146dd9b640 1 -- 172.21.15.6:0/3723795017 learned_addr learned my addr 172.21.15.6:0/3723795017 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:18:33.023 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.023+0000 7f146d59a640 1 -- 172.21.15.6:0/3723795017 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 141659255 0 0) 0x7f14700ab0e0 con 0x7f14700b44b0 2024-03-25T19:18:33.023 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.023+0000 7f146d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:18:33.023 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.023+0000 7f146d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:33.023 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.023+0000 7f146d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:33.023 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.023+0000 7f146d59a640 10 monclient(hunting): my global_id is 4423 2024-03-25T19:18:33.023 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.023+0000 7f146d59a640 1 -- 172.21.15.6:0/3723795017 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1458003520 con 0x7f14700b44b0 2024-03-25T19:18:33.040 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.039+0000 7f146d59a640 1 -- 172.21.15.6:0/3723795017 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2798675103 0 0) 0x7f14700a96f0 con 0x7f14700b4e70 2024-03-25T19:18:33.040 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.039+0000 7f146d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:18:33.040 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.039+0000 7f146d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:33.040 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.039+0000 7f146d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:33.040 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.039+0000 7f146d59a640 10 monclient(hunting): my global_id is 4419 2024-03-25T19:18:33.040 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.039+0000 7f146d59a640 1 -- 172.21.15.6:0/3723795017 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14700ab0e0 con 0x7f14700b4e70 2024-03-25T19:18:33.050 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.050+0000 7f146d59a640 1 -- 172.21.15.6:0/3723795017 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 577267228 0 0) 0x7f1458003520 con 0x7f14700b44b0 2024-03-25T19:18:33.050 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.050+0000 7f146d59a640 1 -- 172.21.15.6:0/3723795017 >> v1:172.21.15.6:6790/0 conn(0x7f14700b1b00 legacy=0x7f147014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:33.050 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.050+0000 7f146d59a640 1 -- 172.21.15.6:0/3723795017 >> v1:172.21.15.6:6789/0 conn(0x7f14700b4e70 legacy=0x7f147013f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:33.050 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.050+0000 7f146d59a640 10 monclient: _finish_hunting 0 2024-03-25T19:18:33.050 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.050+0000 7f146d59a640 1 monclient: found mon.noname-c 2024-03-25T19:18:33.050 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.050+0000 7f146d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:18:33.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.050+0000 7f146d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:18:33.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.050+0000 7f146d59a640 1 -- 172.21.15.6:0/3723795017 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14700a9b80 con 0x7f14700b44b0 2024-03-25T19:18:33.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.050+0000 7f146d59a640 10 monclient: _finish_auth 0 2024-03-25T19:18:33.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.050+0000 7f146d59a640 10 monclient: _check_auth_tickets 2024-03-25T19:18:33.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.050+0000 7f146d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:18:33.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.050+0000 7f146d59a640 1 -- 172.21.15.6:0/3723795017 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f1464001690 con 0x7f14700b44b0 2024-03-25T19:18:33.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.050+0000 7f1474ef6640 5 monclient: authenticate success, global_id 4423 2024-03-25T19:18:33.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.050+0000 7f1474ef6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:18:33.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.050+0000 7f146d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:18:33.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.050+0000 7f146d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:18:33.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.050+0000 7f146d59a640 10 monclient: dump: 2024-03-25T19:18:33.051 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:18:33.051 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:33.052 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:33.052 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:33.052 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:18:33.052 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:33.052 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:18:33.052 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:18:33.052 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:18:33.052 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:33.052 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.050+0000 7f1474ef6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:18:33.060 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.060+0000 7f146d59a640 1 -- 172.21.15.6:0/3723795017 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14640017f0 con 0x7f14700b44b0 2024-03-25T19:18:33.060 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.060+0000 7f146d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:18:33.060 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.060+0000 7f146d59a640 1 -- 172.21.15.6:0/3723795017 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f1464001c00 con 0x7f14700b44b0 2024-03-25T19:18:33.060 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.060+0000 7f146d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:18:33.060 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.060+0000 7f146d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:18:33.060 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.060+0000 7f146d59a640 10 monclient: dump: 2024-03-25T19:18:33.060 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:18:33.060 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:33.060 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:33.060 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:33.060 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:18:33.060 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:33.060 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:18:33.061 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:18:33.061 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:18:33.061 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:33.061 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.060+0000 7f1474ef6640 10 monclient: get_monmap_and_config success 2024-03-25T19:18:33.061 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.060+0000 7f1474ef6640 4 set_mon_vals no callback set 2024-03-25T19:18:33.062 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.060+0000 7f1474ef6640 10 monclient: shutdown 2024-03-25T19:18:33.062 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.060+0000 7f1474ef6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:18:33.062 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.060+0000 7f1474ef6640 1 -- 172.21.15.6:0/3723795017 >> v1:172.21.15.192:6789/0 conn(0x7f14700b44b0 legacy=0x7f1470145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:33.062 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.060+0000 7f1474ef6640 1 -- 172.21.15.6:0/3723795017 shutdown_connections 2024-03-25T19:18:33.062 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.060+0000 7f1474ef6640 1 -- 172.21.15.6:0/3723795017 >> 172.21.15.6:0/3723795017 conn(0x7f14700617a0 msgr2=0x7f1470061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:18:33.063 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.061+0000 7f1474ef6640 1 -- 172.21.15.6:0/3723795017 shutdown_connections 2024-03-25T19:18:33.063 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.061+0000 7f1474ef6640 1 -- 172.21.15.6:0/3723795017 wait complete. 2024-03-25T19:18:33.063 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.061+0000 7f1474ef6640 10 monclient: build_initial_monmap 2024-03-25T19:18:33.063 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.062+0000 7f1474ef6640 10 monclient: monmap: 2024-03-25T19:18:33.063 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:18:33.063 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:18:33.063 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:18:33.063 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:18:33.063 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:18:33.063 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:18:33.063 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:18:33.063 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:18:33.063 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:18:33.063 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:33.063 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.062+0000 7f1474ef6640 1 librados: starting msgr at 2024-03-25T19:18:33.064 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.062+0000 7f1474ef6640 1 librados: starting objecter 2024-03-25T19:18:33.064 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.062+0000 7f1474ef6640 1 Processor -- start 2024-03-25T19:18:33.064 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.062+0000 7f1474ef6640 1 -- start start 2024-03-25T19:18:33.064 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.062+0000 7f1474ef6640 1 librados: setting wanted keys 2024-03-25T19:18:33.064 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.062+0000 7f1474ef6640 1 librados: calling monclient init 2024-03-25T19:18:33.064 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.062+0000 7f1474ef6640 10 monclient: init 2024-03-25T19:18:33.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.063+0000 7f1474ef6640 10 monclient: _reopen_session rank -1 2024-03-25T19:18:33.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.063+0000 7f1474ef6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:18:33.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.063+0000 7f1474ef6640 10 monclient(hunting): picked mon.noname-b con 0x7f14700b1b00 addr v1:172.21.15.192:6789/0 2024-03-25T19:18:33.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.063+0000 7f1474ef6640 10 monclient(hunting): picked mon.noname-c con 0x7f14700b44b0 addr v1:172.21.15.6:6790/0 2024-03-25T19:18:33.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.063+0000 7f1474ef6640 10 monclient(hunting): picked mon.noname-a con 0x7f14700b4e70 addr v1:172.21.15.6:6789/0 2024-03-25T19:18:33.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.063+0000 7f1474ef6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1470131c30 con 0x7f14700b4e70 2024-03-25T19:18:33.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.063+0000 7f1474ef6640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1470130880 con 0x7f14700b1b00 2024-03-25T19:18:33.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.063+0000 7f1474ef6640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f147012e730 con 0x7f14700b44b0 2024-03-25T19:18:33.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.063+0000 7f1474ef6640 10 monclient(hunting): _renew_subs 2024-03-25T19:18:33.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.063+0000 7f1474ef6640 10 monclient(hunting): authenticate will time out at 6217.303321s 2024-03-25T19:18:33.067 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.067+0000 7f146e59c640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f14700b1b00 0x7f147012f7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:39510/0 (socket says 172.21.15.6:39510) 2024-03-25T19:18:33.067 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.067+0000 7f146e59c640 1 -- 172.21.15.6:0/3309874390 learned_addr learned my addr 172.21.15.6:0/3309874390 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:18:33.068 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.068+0000 7f145effd640 10 client.?.objecter ms_handle_connect 0x7f14700b1b00 2024-03-25T19:18:33.068 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.068+0000 7f145effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:18:33.079 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.079+0000 7f145effd640 10 client.?.objecter ms_handle_connect 0x7f14700b4e70 2024-03-25T19:18:33.079 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.079+0000 7f145effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:18:33.080 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.080+0000 7f145effd640 10 client.?.objecter ms_handle_connect 0x7f14700b44b0 2024-03-25T19:18:33.080 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.080+0000 7f145effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:18:33.084 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.084+0000 7f145effd640 1 -- 172.21.15.6:0/3309874390 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1376567135 0 0) 0x7f1470130880 con 0x7f14700b1b00 2024-03-25T19:18:33.084 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.084+0000 7f145effd640 10 monclient(hunting): _negotiate 2024-03-25T19:18:33.085 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.084+0000 7f145effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:33.085 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.084+0000 7f145effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:33.085 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.084+0000 7f145effd640 10 monclient(hunting): my global_id is 4426 2024-03-25T19:18:33.085 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.084+0000 7f145effd640 1 -- 172.21.15.6:0/3309874390 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1450004eb0 con 0x7f14700b1b00 2024-03-25T19:18:33.091 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.091+0000 7f145effd640 1 -- 172.21.15.6:0/3309874390 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 213751514 0 0) 0x7f1470131c30 con 0x7f14700b4e70 2024-03-25T19:18:33.091 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.091+0000 7f145effd640 10 monclient(hunting): _negotiate 2024-03-25T19:18:33.091 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.091+0000 7f145effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:33.091 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.091+0000 7f145effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:33.091 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.091+0000 7f145effd640 10 monclient(hunting): my global_id is 4422 2024-03-25T19:18:33.092 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.091+0000 7f145effd640 1 -- 172.21.15.6:0/3309874390 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1470130880 con 0x7f14700b4e70 2024-03-25T19:18:33.092 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.091+0000 7f145effd640 1 -- 172.21.15.6:0/3309874390 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 742580931 0 0) 0x7f147012e730 con 0x7f14700b44b0 2024-03-25T19:18:33.092 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.091+0000 7f145effd640 10 monclient(hunting): _negotiate 2024-03-25T19:18:33.092 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.091+0000 7f145effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:33.092 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.091+0000 7f145effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:33.092 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.091+0000 7f145effd640 10 monclient(hunting): my global_id is 4424 2024-03-25T19:18:33.092 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.091+0000 7f145effd640 1 -- 172.21.15.6:0/3309874390 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1470131c30 con 0x7f14700b44b0 2024-03-25T19:18:33.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.098+0000 7f145effd640 1 -- 172.21.15.6:0/3309874390 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3209288147 0 0) 0x7f1450004eb0 con 0x7f14700b1b00 2024-03-25T19:18:33.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.098+0000 7f145effd640 1 -- 172.21.15.6:0/3309874390 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f147012e730 con 0x7f14700b1b00 2024-03-25T19:18:33.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.099+0000 7f145effd640 1 -- 172.21.15.6:0/3309874390 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f1460001990 con 0x7f14700b1b00 2024-03-25T19:18:33.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.099+0000 7f145effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:18:33.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.099+0000 7f145effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:18:33.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.099+0000 7f145effd640 10 monclient(hunting): dump: 2024-03-25T19:18:33.099 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:18:33.099 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:33.099 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:33.099 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:33.099 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:18:33.099 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:33.100 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:18:33.100 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:18:33.100 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:18:33.100 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:33.100 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.099+0000 7f145effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:18:33.107 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.106+0000 7f145effd640 1 -- 172.21.15.6:0/3309874390 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 834897917 0 0) 0x7f1470130880 con 0x7f14700b4e70 2024-03-25T19:18:33.107 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.106+0000 7f145effd640 1 -- 172.21.15.6:0/3309874390 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1450004eb0 con 0x7f14700b4e70 2024-03-25T19:18:33.107 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.106+0000 7f145effd640 1 -- 172.21.15.6:0/3309874390 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f1468001bb0 con 0x7f14700b4e70 2024-03-25T19:18:33.107 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.106+0000 7f145effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:18:33.107 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.106+0000 7f145effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:18:33.107 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.106+0000 7f145effd640 10 monclient(hunting): dump: 2024-03-25T19:18:33.107 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:18:33.107 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:33.107 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:33.107 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:33.107 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:18:33.107 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:33.107 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:18:33.108 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:18:33.108 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:18:33.108 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:33.109 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.109+0000 7f145effd640 1 -- 172.21.15.6:0/3309874390 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3912436256 0 0) 0x7f1470131c30 con 0x7f14700b44b0 2024-03-25T19:18:33.109 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.109+0000 7f145effd640 1 -- 172.21.15.6:0/3309874390 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1470130880 con 0x7f14700b44b0 2024-03-25T19:18:33.109 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.109+0000 7f145effd640 1 -- 172.21.15.6:0/3309874390 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f1464001b00 con 0x7f14700b44b0 2024-03-25T19:18:33.109 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.109+0000 7f145effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:18:33.109 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.109+0000 7f145effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:18:33.109 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.109+0000 7f145effd640 10 monclient(hunting): dump: 2024-03-25T19:18:33.109 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:18:33.109 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:33.109 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:33.109 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:33.109 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:18:33.110 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:33.110 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:18:33.110 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:18:33.110 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:18:33.110 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:33.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.117+0000 7f145effd640 1 -- 172.21.15.6:0/3309874390 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2534468983 0 0) 0x7f147012e730 con 0x7f14700b1b00 2024-03-25T19:18:33.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.117+0000 7f145effd640 1 -- 172.21.15.6:0/3309874390 >> v1:172.21.15.6:6790/0 conn(0x7f14700b44b0 legacy=0x7f147012ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:33.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.117+0000 7f145effd640 1 -- 172.21.15.6:0/3309874390 >> v1:172.21.15.6:6789/0 conn(0x7f14700b4e70 legacy=0x7f147012e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:33.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.117+0000 7f145effd640 10 monclient: _finish_hunting 0 2024-03-25T19:18:33.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.117+0000 7f145effd640 1 monclient: found mon.b 2024-03-25T19:18:33.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.117+0000 7f145effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:18:33.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.117+0000 7f145effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:18:33.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.117+0000 7f145effd640 1 -- 172.21.15.6:0/3309874390 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f147012e8c0 con 0x7f14700b1b00 2024-03-25T19:18:33.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.117+0000 7f145effd640 10 monclient: _finish_auth 0 2024-03-25T19:18:33.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.117+0000 7f145effd640 10 monclient: _check_auth_tickets 2024-03-25T19:18:33.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.117+0000 7f145effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:18:33.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.117+0000 7f1474ef6640 5 monclient: authenticate success, global_id 4426 2024-03-25T19:18:33.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.117+0000 7f1474ef6640 10 monclient: _renew_subs 2024-03-25T19:18:33.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.117+0000 7f1474ef6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:18:33.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.117+0000 7f1474ef6640 1 -- 172.21.15.6:0/3309874390 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14701dad50 con 0x7f14700b1b00 2024-03-25T19:18:33.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.117+0000 7f1474ef6640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:18:33.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.117+0000 7f1474ef6640 10 monclient: _renew_subs 2024-03-25T19:18:33.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.117+0000 7f1474ef6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:18:33.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.117+0000 7f1474ef6640 1 -- 172.21.15.6:0/3309874390 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f14701db1e0 con 0x7f14700b1b00 2024-03-25T19:18:33.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.117+0000 7f1474ef6640 1 librados: init done 2024-03-25T19:18:33.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.118+0000 7f1474ef6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:18:33.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.118+0000 7f1474ef6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:18:33.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.118+0000 7f1474ef6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:18:33.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.118+0000 7f1474ef6640 1 -- 172.21.15.6:0/3309874390 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1470130680 con 0x7f14700b1b00 2024-03-25T19:18:33.143 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.143+0000 7f145effd640 1 -- 172.21.15.6:0/3309874390 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1460003610 con 0x7f14700b1b00 2024-03-25T19:18:33.143 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.143+0000 7f145effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:18:33.143 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.143+0000 7f145effd640 1 -- 172.21.15.6:0/3309874390 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f14600039a0 con 0x7f14700b1b00 2024-03-25T19:18:33.143 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.143+0000 7f145effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:18:33.143 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.143+0000 7f145effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:18:33.143 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.143+0000 7f145effd640 10 monclient: dump: 2024-03-25T19:18:33.144 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:18:33.144 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:33.144 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:33.144 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:33.144 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:18:33.144 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:33.144 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:18:33.144 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:18:33.144 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:18:33.144 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:33.144 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.143+0000 7f146cd99640 4 set_mon_vals no callback set 2024-03-25T19:18:33.145 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.144+0000 7f145effd640 1 -- 172.21.15.6:0/3309874390 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f1460003bc0 con 0x7f14700b1b00 2024-03-25T19:18:33.145 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.145+0000 7f146dd9b640 10 monclient: get_auth_request con 0x7f145006e130 auth_method 0 2024-03-25T19:18:33.145 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.145+0000 7f145effd640 10 client.4426.objecter ms_handle_connect 0x7f145006e130 2024-03-25T19:18:33.397 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.397+0000 7f145effd640 1 -- 172.21.15.6:0/3309874390 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f14600547e0 con 0x7f14700b1b00 2024-03-25T19:18:33.397 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.397+0000 7f145effd640 10 client.4426.objecter ms_dispatch 0x7f1470000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:18:33.397 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.397+0000 7f145effd640 3 client.4426.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:18:33.398 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.397+0000 7f145effd640 3 client.4426.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:18:33.398 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.397+0000 7f145effd640 20 client.4426.objecter dump_active .. 0 homeless 2024-03-25T19:18:33.398 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.398+0000 7f145effd640 1 -- 172.21.15.6:0/3309874390 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f14600502f0 con 0x7f14700b1b00 2024-03-25T19:18:33.398 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.398+0000 7f145effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:18:33.398 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.398+0000 7f145effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:18:33.646 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.646+0000 7f1474ef6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:18:33.646 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.646+0000 7f1474ef6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:18:33.647 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.646+0000 7f1474ef6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:18:33.647 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.646+0000 7f1474ef6640 1 -- 172.21.15.6:0/3309874390 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f147004f730 con 0x7f14700b1b00 2024-03-25T19:18:33.682 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.682+0000 7f145effd640 1 -- 172.21.15.6:0/3309874390 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f1460052220 con 0x7f14700b1b00 2024-03-25T19:18:33.682 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.682+0000 7f145effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:18:33.682 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.682+0000 7f145effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:18:33.682 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:18:33.682 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:18:33.685 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.684+0000 7f1474ef6640 10 librados: watch_flush enter 2024-03-25T19:18:33.685 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.684+0000 7f1474ef6640 10 librados: watch_flush exit 2024-03-25T19:18:33.685 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.684+0000 7f1474ef6640 20 client.4426.objecter shutdown clearing up homeless session... 2024-03-25T19:18:33.685 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.684+0000 7f1474ef6640 10 client.4426.objecter successfully canceled tick 2024-03-25T19:18:33.685 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.684+0000 7f1474ef6640 1 -- 172.21.15.6:0/3309874390 >> v1:172.21.15.6:6800/2936652650 conn(0x7f145006e130 legacy=0x7f14500705f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:33.685 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.684+0000 7f1474ef6640 10 monclient: shutdown 2024-03-25T19:18:33.685 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.684+0000 7f1474ef6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:18:33.685 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.684+0000 7f1474ef6640 1 -- 172.21.15.6:0/3309874390 >> v1:172.21.15.192:6789/0 conn(0x7f14700b1b00 legacy=0x7f147012f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:33.685 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.684+0000 7f1474ef6640 1 -- 172.21.15.6:0/3309874390 shutdown_connections 2024-03-25T19:18:33.686 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.684+0000 7f1474ef6640 1 -- 172.21.15.6:0/3309874390 >> 172.21.15.6:0/3309874390 conn(0x7f14700617a0 msgr2=0x7f14700b6510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:18:33.686 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.684+0000 7f1474ef6640 1 -- 172.21.15.6:0/3309874390 shutdown_connections 2024-03-25T19:18:33.686 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.684+0000 7f1474ef6640 1 -- 172.21.15.6:0/3309874390 wait complete. 2024-03-25T19:18:33.686 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:33.684+0000 7f1474ef6640 1 librados: shutdown 2024-03-25T19:18:33.696 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:18:40.502 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:18:40.619 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.619+0000 7fce3dd5d640 10 monclient: get_monmap_and_config 2024-03-25T19:18:40.619 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.619+0000 7fce3dd5d640 10 monclient: build_initial_monmap 2024-03-25T19:18:40.619 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.619+0000 7fce3dd5d640 10 monclient: monmap: 2024-03-25T19:18:40.619 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:18:40.619 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:40.619 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:18:40.620315+0000 2024-03-25T19:18:40.619 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:18:40.620315+0000 2024-03-25T19:18:40.619 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:18:40.620 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:40.620 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:18:40.620 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:18:40.620 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:18:40.620 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:40.621 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.620+0000 7fce3dd5d640 1 Processor -- start 2024-03-25T19:18:40.621 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.620+0000 7fce3dd5d640 1 -- start start 2024-03-25T19:18:40.621 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.620+0000 7fce3dd5d640 10 monclient: init 2024-03-25T19:18:40.621 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.620+0000 7fce3dd5d640 10 monclient: _reopen_session rank -1 2024-03-25T19:18:40.621 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.620+0000 7fce3dd5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:18:40.621 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.620+0000 7fce3dd5d640 10 monclient(hunting): picked mon.noname-a con 0x7fce380b4e70 addr v1:172.21.15.6:6789/0 2024-03-25T19:18:40.621 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.620+0000 7fce3dd5d640 10 monclient(hunting): picked mon.noname-c con 0x7fce380b44b0 addr v1:172.21.15.192:6789/0 2024-03-25T19:18:40.621 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.620+0000 7fce3dd5d640 10 monclient(hunting): picked mon.noname-b con 0x7fce380b1b00 addr v1:172.21.15.6:6790/0 2024-03-25T19:18:40.621 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.620+0000 7fce3dd5d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce380a96f0 con 0x7fce380b4e70 2024-03-25T19:18:40.621 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.620+0000 7fce3dd5d640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce380ab0e0 con 0x7fce380b44b0 2024-03-25T19:18:40.622 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.620+0000 7fce3dd5d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce38145f70 con 0x7fce380b1b00 2024-03-25T19:18:40.622 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.620+0000 7fce3dd5d640 10 monclient(hunting): _renew_subs 2024-03-25T19:18:40.622 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.620+0000 7fce3dd5d640 10 monclient(hunting): authenticate will time out at 6224.860720s 2024-03-25T19:18:40.632 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.632+0000 7fce37fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fce380b1b00 0x7fce3814dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:56738/0 (socket says 172.21.15.6:56738) 2024-03-25T19:18:40.632 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.632+0000 7fce37fff640 1 -- 172.21.15.6:0/1930346649 learned_addr learned my addr 172.21.15.6:0/1930346649 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:18:40.642 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.642+0000 7fce367fc640 1 -- 172.21.15.6:0/1930346649 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 446445761 0 0) 0x7fce38145f70 con 0x7fce380b1b00 2024-03-25T19:18:40.642 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.642+0000 7fce367fc640 10 monclient(hunting): _negotiate 2024-03-25T19:18:40.642 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.642+0000 7fce367fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:40.642 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.642+0000 7fce367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:40.643 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.642+0000 7fce367fc640 10 monclient(hunting): my global_id is 4427 2024-03-25T19:18:40.643 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.642+0000 7fce367fc640 1 -- 172.21.15.6:0/1930346649 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce20003520 con 0x7fce380b1b00 2024-03-25T19:18:40.644 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.644+0000 7fce367fc640 1 -- 172.21.15.6:0/1930346649 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3816285044 0 0) 0x7fce380a96f0 con 0x7fce380b4e70 2024-03-25T19:18:40.644 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.644+0000 7fce367fc640 10 monclient(hunting): _negotiate 2024-03-25T19:18:40.645 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.644+0000 7fce367fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:40.645 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.644+0000 7fce367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:40.645 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.644+0000 7fce367fc640 10 monclient(hunting): my global_id is 4425 2024-03-25T19:18:40.645 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.644+0000 7fce367fc640 1 -- 172.21.15.6:0/1930346649 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce38145f70 con 0x7fce380b4e70 2024-03-25T19:18:40.647 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.647+0000 7fce367fc640 1 -- 172.21.15.6:0/1930346649 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2517361896 0 0) 0x7fce380ab0e0 con 0x7fce380b44b0 2024-03-25T19:18:40.647 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.647+0000 7fce367fc640 10 monclient(hunting): _negotiate 2024-03-25T19:18:40.647 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.647+0000 7fce367fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:40.647 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.647+0000 7fce367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:40.647 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.647+0000 7fce367fc640 10 monclient(hunting): my global_id is 4429 2024-03-25T19:18:40.648 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.647+0000 7fce367fc640 1 -- 172.21.15.6:0/1930346649 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce380a96f0 con 0x7fce380b44b0 2024-03-25T19:18:40.653 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.652+0000 7fce367fc640 1 -- 172.21.15.6:0/1930346649 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2499933651 0 0) 0x7fce20003520 con 0x7fce380b1b00 2024-03-25T19:18:40.653 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.652+0000 7fce367fc640 1 -- 172.21.15.6:0/1930346649 >> v1:172.21.15.192:6789/0 conn(0x7fce380b44b0 legacy=0x7fce38145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:40.653 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.652+0000 7fce367fc640 1 -- 172.21.15.6:0/1930346649 >> v1:172.21.15.6:6789/0 conn(0x7fce380b4e70 legacy=0x7fce3813f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:40.653 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.652+0000 7fce367fc640 10 monclient: _finish_hunting 0 2024-03-25T19:18:40.653 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.652+0000 7fce367fc640 1 monclient: found mon.noname-b 2024-03-25T19:18:40.653 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.653+0000 7fce367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:18:40.653 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.653+0000 7fce367fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-03-25T19:18:40.653 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.653+0000 7fce367fc640 1 -- 172.21.15.6:0/1930346649 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce380a9b80 con 0x7fce380b1b00 2024-03-25T19:18:40.653 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.653+0000 7fce367fc640 10 monclient: _finish_auth 0 2024-03-25T19:18:40.653 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.653+0000 7fce367fc640 10 monclient: _check_auth_tickets 2024-03-25T19:18:40.653 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.653+0000 7fce367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:18:40.654 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.653+0000 7fce367fc640 1 -- 172.21.15.6:0/1930346649 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fce30001a10 con 0x7fce380b1b00 2024-03-25T19:18:40.654 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.653+0000 7fce367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:18:40.654 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.653+0000 7fce367fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:18:40.654 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.653+0000 7fce367fc640 10 monclient: dump: 2024-03-25T19:18:40.654 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:18:40.654 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:40.654 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:40.654 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:40.654 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:18:40.654 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:40.654 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:18:40.654 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:18:40.654 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:18:40.655 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:40.655 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.653+0000 7fce3dd5d640 5 monclient: authenticate success, global_id 4427 2024-03-25T19:18:40.655 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.653+0000 7fce3dd5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:18:40.663 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.662+0000 7fce367fc640 1 -- 172.21.15.6:0/1930346649 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce30001b70 con 0x7fce380b1b00 2024-03-25T19:18:40.663 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.663+0000 7fce367fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:18:40.663 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.663+0000 7fce367fc640 1 -- 172.21.15.6:0/1930346649 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fce30001f80 con 0x7fce380b1b00 2024-03-25T19:18:40.663 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.663+0000 7fce3dd5d640 10 monclient: get_monmap_and_config success 2024-03-25T19:18:40.663 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.663+0000 7fce3dd5d640 4 set_mon_vals no callback set 2024-03-25T19:18:40.663 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.663+0000 7fce367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:18:40.663 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.663+0000 7fce367fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:18:40.663 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.663+0000 7fce367fc640 10 monclient: dump: 2024-03-25T19:18:40.663 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:18:40.663 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:40.663 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:40.663 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:40.663 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:18:40.664 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:40.664 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:18:40.664 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:18:40.664 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:18:40.664 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:40.665 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.663+0000 7fce3dd5d640 10 monclient: shutdown 2024-03-25T19:18:40.665 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.663+0000 7fce3dd5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:18:40.665 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.663+0000 7fce3dd5d640 1 -- 172.21.15.6:0/1930346649 >> v1:172.21.15.6:6790/0 conn(0x7fce380b1b00 legacy=0x7fce3814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:40.665 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.663+0000 7fce3dd5d640 1 -- 172.21.15.6:0/1930346649 shutdown_connections 2024-03-25T19:18:40.665 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.663+0000 7fce3dd5d640 1 -- 172.21.15.6:0/1930346649 >> 172.21.15.6:0/1930346649 conn(0x7fce380617a0 msgr2=0x7fce38061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:18:40.665 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.663+0000 7fce3dd5d640 1 -- 172.21.15.6:0/1930346649 shutdown_connections 2024-03-25T19:18:40.666 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.664+0000 7fce3dd5d640 1 -- 172.21.15.6:0/1930346649 wait complete. 2024-03-25T19:18:40.666 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.664+0000 7fce3dd5d640 10 monclient: build_initial_monmap 2024-03-25T19:18:40.666 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.664+0000 7fce3dd5d640 10 monclient: monmap: 2024-03-25T19:18:40.666 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:18:40.666 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:18:40.666 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:18:40.666 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:18:40.666 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:18:40.666 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:18:40.666 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:18:40.666 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:18:40.666 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:18:40.666 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:40.666 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.664+0000 7fce3dd5d640 1 librados: starting msgr at 2024-03-25T19:18:40.666 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.664+0000 7fce3dd5d640 1 librados: starting objecter 2024-03-25T19:18:40.667 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.665+0000 7fce3dd5d640 1 Processor -- start 2024-03-25T19:18:40.667 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.665+0000 7fce3dd5d640 1 -- start start 2024-03-25T19:18:40.667 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.665+0000 7fce3dd5d640 1 librados: setting wanted keys 2024-03-25T19:18:40.667 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.665+0000 7fce3dd5d640 1 librados: calling monclient init 2024-03-25T19:18:40.667 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.665+0000 7fce3dd5d640 10 monclient: init 2024-03-25T19:18:40.668 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.666+0000 7fce3dd5d640 10 monclient: _reopen_session rank -1 2024-03-25T19:18:40.668 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.666+0000 7fce3dd5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:18:40.668 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.666+0000 7fce3dd5d640 10 monclient(hunting): picked mon.noname-a con 0x7fce380b1b00 addr v1:172.21.15.6:6789/0 2024-03-25T19:18:40.668 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.666+0000 7fce3dd5d640 10 monclient(hunting): picked mon.noname-b con 0x7fce380b44b0 addr v1:172.21.15.192:6789/0 2024-03-25T19:18:40.668 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.666+0000 7fce3dd5d640 10 monclient(hunting): picked mon.noname-c con 0x7fce380b4e70 addr v1:172.21.15.6:6790/0 2024-03-25T19:18:40.668 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.666+0000 7fce3dd5d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce38127e30 con 0x7fce380b1b00 2024-03-25T19:18:40.668 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.666+0000 7fce3dd5d640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce3813c400 con 0x7fce380b44b0 2024-03-25T19:18:40.668 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.666+0000 7fce3dd5d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce3813c6e0 con 0x7fce380b4e70 2024-03-25T19:18:40.668 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.666+0000 7fce3dd5d640 10 monclient(hunting): _renew_subs 2024-03-25T19:18:40.668 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.666+0000 7fce3dd5d640 10 monclient(hunting): authenticate will time out at 6224.906188s 2024-03-25T19:18:40.680 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.680+0000 7fce37fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fce380b4e70 0x7fce381e1e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:56786/0 (socket says 172.21.15.6:56786) 2024-03-25T19:18:40.680 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.680+0000 7fce37fff640 1 -- 172.21.15.6:0/2479349481 learned_addr learned my addr 172.21.15.6:0/2479349481 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:18:40.684 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.684+0000 7fce1ffff640 10 client.?.objecter ms_handle_connect 0x7fce380b44b0 2024-03-25T19:18:40.684 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.684+0000 7fce1ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:18:40.685 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.685+0000 7fce1ffff640 10 client.?.objecter ms_handle_connect 0x7fce380b4e70 2024-03-25T19:18:40.685 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.685+0000 7fce1ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:18:40.689 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.689+0000 7fce1ffff640 10 client.?.objecter ms_handle_connect 0x7fce380b1b00 2024-03-25T19:18:40.689 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.689+0000 7fce1ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:18:40.690 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.690+0000 7fce1ffff640 1 -- 172.21.15.6:0/2479349481 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1686040862 0 0) 0x7fce3813c400 con 0x7fce380b44b0 2024-03-25T19:18:40.690 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.690+0000 7fce1ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:18:40.690 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.690+0000 7fce1ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:40.690 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.690+0000 7fce1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:40.690 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.690+0000 7fce1ffff640 10 monclient(hunting): my global_id is 4432 2024-03-25T19:18:40.690 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.690+0000 7fce1ffff640 1 -- 172.21.15.6:0/2479349481 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce24004eb0 con 0x7fce380b44b0 2024-03-25T19:18:40.696 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.697+0000 7fce1ffff640 1 -- 172.21.15.6:0/2479349481 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2006295642 0 0) 0x7fce3813c6e0 con 0x7fce380b4e70 2024-03-25T19:18:40.697 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.697+0000 7fce1ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:18:40.697 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.697+0000 7fce1ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:40.697 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.697+0000 7fce1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:40.697 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.697+0000 7fce1ffff640 10 monclient(hunting): my global_id is 4430 2024-03-25T19:18:40.697 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.697+0000 7fce1ffff640 1 -- 172.21.15.6:0/2479349481 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce3813c400 con 0x7fce380b4e70 2024-03-25T19:18:40.703 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.703+0000 7fce1ffff640 1 -- 172.21.15.6:0/2479349481 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 534238544 0 0) 0x7fce38127e30 con 0x7fce380b1b00 2024-03-25T19:18:40.703 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.703+0000 7fce1ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:18:40.703 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.703+0000 7fce1ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:40.703 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.703+0000 7fce1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:40.703 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.703+0000 7fce1ffff640 10 monclient(hunting): my global_id is 4428 2024-03-25T19:18:40.703 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.703+0000 7fce1ffff640 1 -- 172.21.15.6:0/2479349481 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce3813c6e0 con 0x7fce380b1b00 2024-03-25T19:18:40.704 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.703+0000 7fce1ffff640 1 -- 172.21.15.6:0/2479349481 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3348195495 0 0) 0x7fce3813c400 con 0x7fce380b4e70 2024-03-25T19:18:40.704 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.703+0000 7fce1ffff640 1 -- 172.21.15.6:0/2479349481 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce38127e30 con 0x7fce380b4e70 2024-03-25T19:18:40.704 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.703+0000 7fce1ffff640 1 -- 172.21.15.6:0/2479349481 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fce30001e80 con 0x7fce380b4e70 2024-03-25T19:18:40.704 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.703+0000 7fce1ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:18:40.704 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.703+0000 7fce1ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:18:40.704 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.703+0000 7fce1ffff640 10 monclient(hunting): dump: 2024-03-25T19:18:40.705 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:18:40.705 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:40.705 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:40.705 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:40.705 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:18:40.705 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:40.705 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:18:40.705 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:18:40.705 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:18:40.705 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:40.705 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.703+0000 7fce1ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:18:40.707 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.706+0000 7fce1ffff640 1 -- 172.21.15.6:0/2479349481 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 871862780 0 0) 0x7fce24004eb0 con 0x7fce380b44b0 2024-03-25T19:18:40.707 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.707+0000 7fce1ffff640 1 -- 172.21.15.6:0/2479349481 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce3813c400 con 0x7fce380b44b0 2024-03-25T19:18:40.707 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.707+0000 7fce1ffff640 1 -- 172.21.15.6:0/2479349481 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fce2c0018f0 con 0x7fce380b44b0 2024-03-25T19:18:40.707 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.707+0000 7fce1ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:18:40.707 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.707+0000 7fce1ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:18:40.707 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.707+0000 7fce1ffff640 10 monclient(hunting): dump: 2024-03-25T19:18:40.707 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:18:40.707 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:40.707 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:40.707 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:40.707 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:18:40.707 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:40.707 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:18:40.707 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:18:40.708 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:18:40.708 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:40.713 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.712+0000 7fce1ffff640 1 -- 172.21.15.6:0/2479349481 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2812230881 0 0) 0x7fce3813c400 con 0x7fce380b44b0 2024-03-25T19:18:40.713 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.712+0000 7fce1ffff640 1 -- 172.21.15.6:0/2479349481 >> v1:172.21.15.6:6790/0 conn(0x7fce380b4e70 legacy=0x7fce381e1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:40.713 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.712+0000 7fce1ffff640 1 -- 172.21.15.6:0/2479349481 >> v1:172.21.15.6:6789/0 conn(0x7fce380b1b00 legacy=0x7fce3813b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:40.713 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.713+0000 7fce1ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:18:40.714 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.713+0000 7fce1ffff640 1 monclient: found mon.b 2024-03-25T19:18:40.714 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.713+0000 7fce1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:18:40.714 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.713+0000 7fce1ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:18:40.714 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.713+0000 7fce1ffff640 1 -- 172.21.15.6:0/2479349481 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce3813c9c0 con 0x7fce380b44b0 2024-03-25T19:18:40.714 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.713+0000 7fce1ffff640 10 monclient: _finish_auth 0 2024-03-25T19:18:40.714 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.713+0000 7fce1ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:18:40.714 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.713+0000 7fce1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:18:40.714 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.713+0000 7fce3dd5d640 5 monclient: authenticate success, global_id 4432 2024-03-25T19:18:40.714 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.713+0000 7fce3dd5d640 10 monclient: _renew_subs 2024-03-25T19:18:40.714 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.713+0000 7fce3dd5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:18:40.714 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.713+0000 7fce3dd5d640 1 -- 172.21.15.6:0/2479349481 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce3813cca0 con 0x7fce380b44b0 2024-03-25T19:18:40.714 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.713+0000 7fce3dd5d640 10 client.4432.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:18:40.714 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.713+0000 7fce3dd5d640 10 monclient: _renew_subs 2024-03-25T19:18:40.715 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.713+0000 7fce3dd5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:18:40.715 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.713+0000 7fce3dd5d640 1 -- 172.21.15.6:0/2479349481 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fce3813ba80 con 0x7fce380b44b0 2024-03-25T19:18:40.715 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.713+0000 7fce3dd5d640 1 librados: init done 2024-03-25T19:18:40.715 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.714+0000 7fce3dd5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:18:40.715 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.714+0000 7fce3dd5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:18:40.715 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.714+0000 7fce3dd5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:18:40.715 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.714+0000 7fce3dd5d640 1 -- 172.21.15.6:0/2479349481 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce3804f730 con 0x7fce380b44b0 2024-03-25T19:18:40.993 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.993+0000 7fce1ffff640 1 -- 172.21.15.6:0/2479349481 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce2c003610 con 0x7fce380b44b0 2024-03-25T19:18:40.993 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.993+0000 7fce1ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:18:40.994 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.993+0000 7fce1ffff640 1 -- 172.21.15.6:0/2479349481 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fce2c003920 con 0x7fce380b44b0 2024-03-25T19:18:40.994 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.993+0000 7fce1ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:18:40.994 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.993+0000 7fce1ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:18:40.994 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.993+0000 7fce1ffff640 10 monclient: dump: 2024-03-25T19:18:40.994 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:18:40.994 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:40.994 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:40.994 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:40.994 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:18:40.994 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:40.994 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:18:40.994 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:18:40.994 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:18:40.994 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:40.995 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.993+0000 7fce35ffb640 4 set_mon_vals no callback set 2024-03-25T19:18:40.995 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.994+0000 7fce1ffff640 1 -- 172.21.15.6:0/2479349481 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7fce2c019930 con 0x7fce380b44b0 2024-03-25T19:18:40.995 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.995+0000 7fce377fe640 10 monclient: get_auth_request con 0x7fce2406dfb0 auth_method 0 2024-03-25T19:18:40.995 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.995+0000 7fce1ffff640 10 client.4432.objecter ms_handle_connect 0x7fce2406dfb0 2024-03-25T19:18:40.999 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.998+0000 7fce1ffff640 1 -- 172.21.15.6:0/2479349481 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7fce2c053620 con 0x7fce380b44b0 2024-03-25T19:18:40.999 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.998+0000 7fce1ffff640 10 client.4432.objecter ms_dispatch 0x7fce38000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:18:40.999 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.998+0000 7fce1ffff640 3 client.4432.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:18:40.999 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.998+0000 7fce1ffff640 3 client.4432.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:18:40.999 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.998+0000 7fce1ffff640 20 client.4432.objecter dump_active .. 0 homeless 2024-03-25T19:18:40.999 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.999+0000 7fce1ffff640 1 -- 172.21.15.6:0/2479349481 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fce2c04ef60 con 0x7fce380b44b0 2024-03-25T19:18:40.999 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.999+0000 7fce1ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:18:40.999 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:40.999+0000 7fce1ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:18:41.223 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:41.223+0000 7fce3dd5d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:18:41.223 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:41.223+0000 7fce3dd5d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:18:41.223 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:41.223+0000 7fce3dd5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:18:41.224 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:41.223+0000 7fce3dd5d640 1 -- 172.21.15.6:0/2479349481 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce380aa090 con 0x7fce380b44b0 2024-03-25T19:18:41.255 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:41.255+0000 7fce1ffff640 1 -- 172.21.15.6:0/2479349481 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7fce2c051060 con 0x7fce380b44b0 2024-03-25T19:18:41.255 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:41.255+0000 7fce1ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:18:41.255 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:41.255+0000 7fce1ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:18:41.256 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:18:41.256 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:18:41.258 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:41.257+0000 7fce3dd5d640 10 librados: watch_flush enter 2024-03-25T19:18:41.258 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:41.257+0000 7fce3dd5d640 10 librados: watch_flush exit 2024-03-25T19:18:41.258 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:41.257+0000 7fce3dd5d640 20 client.4432.objecter shutdown clearing up homeless session... 2024-03-25T19:18:41.258 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:41.257+0000 7fce3dd5d640 10 client.4432.objecter successfully canceled tick 2024-03-25T19:18:41.258 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:41.258+0000 7fce3dd5d640 1 -- 172.21.15.6:0/2479349481 >> v1:172.21.15.6:6800/2936652650 conn(0x7fce2406dfb0 legacy=0x7fce24070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:41.258 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:41.258+0000 7fce3dd5d640 10 monclient: shutdown 2024-03-25T19:18:41.258 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:41.258+0000 7fce3dd5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:18:41.259 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:41.258+0000 7fce3dd5d640 1 -- 172.21.15.6:0/2479349481 >> v1:172.21.15.192:6789/0 conn(0x7fce380b44b0 legacy=0x7fce38143a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:41.259 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:41.258+0000 7fce3dd5d640 1 -- 172.21.15.6:0/2479349481 shutdown_connections 2024-03-25T19:18:41.259 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:41.258+0000 7fce3dd5d640 1 -- 172.21.15.6:0/2479349481 >> 172.21.15.6:0/2479349481 conn(0x7fce380617a0 msgr2=0x7fce380b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:18:41.259 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:41.258+0000 7fce3dd5d640 1 -- 172.21.15.6:0/2479349481 shutdown_connections 2024-03-25T19:18:41.259 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:41.258+0000 7fce3dd5d640 1 -- 172.21.15.6:0/2479349481 wait complete. 2024-03-25T19:18:41.259 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:41.258+0000 7fce3dd5d640 1 librados: shutdown 2024-03-25T19:18:41.268 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:18:43.274 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:05:00.621 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:18:43.279 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:05:00.626 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:18:43.328 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:05:00.646 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:18:43.328 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:05:00.675 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:18:43.333 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:05:00.680 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:18:43.335 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:05:00.653 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:18:43.348 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:05:00.695 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:18:43.359 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:05:00.677 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:18:43.375 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:05:00.722 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:18:48.075 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:18:48.189 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.189+0000 7f6da2fac640 10 monclient: get_monmap_and_config 2024-03-25T19:18:48.189 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.189+0000 7f6da2fac640 10 monclient: build_initial_monmap 2024-03-25T19:18:48.190 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.189+0000 7f6da2fac640 10 monclient: monmap: 2024-03-25T19:18:48.190 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:18:48.190 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:48.190 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:18:48.190781+0000 2024-03-25T19:18:48.190 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:18:48.190781+0000 2024-03-25T19:18:48.190 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:18:48.190 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:48.190 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:18:48.190 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:18:48.190 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:18:48.190 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:48.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.190+0000 7f6da2fac640 1 Processor -- start 2024-03-25T19:18:48.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.190+0000 7f6da2fac640 1 -- start start 2024-03-25T19:18:48.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.190+0000 7f6da2fac640 10 monclient: init 2024-03-25T19:18:48.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.191+0000 7f6da2fac640 10 monclient: _reopen_session rank -1 2024-03-25T19:18:48.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.191+0000 7f6da2fac640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:18:48.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.191+0000 7f6da2fac640 10 monclient(hunting): picked mon.noname-a con 0x7f6d9c0b4e70 addr v1:172.21.15.6:6789/0 2024-03-25T19:18:48.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.191+0000 7f6da2fac640 10 monclient(hunting): picked mon.noname-b con 0x7f6d9c0b44b0 addr v1:172.21.15.6:6790/0 2024-03-25T19:18:48.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.191+0000 7f6da2fac640 10 monclient(hunting): picked mon.noname-c con 0x7f6d9c0b1b00 addr v1:172.21.15.192:6789/0 2024-03-25T19:18:48.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.191+0000 7f6da2fac640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9c0a96f0 con 0x7f6d9c0b4e70 2024-03-25T19:18:48.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.191+0000 7f6da2fac640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9c0ab0e0 con 0x7f6d9c0b1b00 2024-03-25T19:18:48.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.191+0000 7f6da2fac640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9c145f70 con 0x7f6d9c0b44b0 2024-03-25T19:18:48.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.191+0000 7f6da2fac640 10 monclient(hunting): _renew_subs 2024-03-25T19:18:48.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.191+0000 7f6da2fac640 10 monclient(hunting): authenticate will time out at 6232.431201s 2024-03-25T19:18:48.205 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.205+0000 7f6d9bfff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f6d9c0b44b0 0x7f6d9c145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:32972/0 (socket says 172.21.15.6:32972) 2024-03-25T19:18:48.205 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.205+0000 7f6d9bfff640 1 -- 172.21.15.6:0/2831991906 learned_addr learned my addr 172.21.15.6:0/2831991906 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:18:48.215 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.215+0000 7f6d9b7fe640 1 -- 172.21.15.6:0/2831991906 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4049389518 0 0) 0x7f6d9c0ab0e0 con 0x7f6d9c0b1b00 2024-03-25T19:18:48.215 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.215+0000 7f6d9b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:18:48.216 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.215+0000 7f6d9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:48.216 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.215+0000 7f6d9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:48.216 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.215+0000 7f6d9b7fe640 10 monclient(hunting): my global_id is 4435 2024-03-25T19:18:48.216 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.215+0000 7f6d9b7fe640 1 -- 172.21.15.6:0/2831991906 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d84003520 con 0x7f6d9c0b1b00 2024-03-25T19:18:48.224 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.224+0000 7f6d9b7fe640 1 -- 172.21.15.6:0/2831991906 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2519960735 0 0) 0x7f6d84003520 con 0x7f6d9c0b1b00 2024-03-25T19:18:48.224 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.224+0000 7f6d9b7fe640 1 -- 172.21.15.6:0/2831991906 >> v1:172.21.15.6:6790/0 conn(0x7f6d9c0b44b0 legacy=0x7f6d9c145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:48.224 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.224+0000 7f6d9b7fe640 1 -- 172.21.15.6:0/2831991906 >> v1:172.21.15.6:6789/0 conn(0x7f6d9c0b4e70 legacy=0x7f6d9c13f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:48.224 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.224+0000 7f6d9b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:18:48.224 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.224+0000 7f6d9b7fe640 1 monclient: found mon.noname-c 2024-03-25T19:18:48.224 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.224+0000 7f6d9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:18:48.225 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.224+0000 7f6d9b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:18:48.225 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.224+0000 7f6d9b7fe640 1 -- 172.21.15.6:0/2831991906 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d9c0a9b80 con 0x7f6d9c0b1b00 2024-03-25T19:18:48.225 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.224+0000 7f6d9b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:18:48.225 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.224+0000 7f6d9b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:18:48.225 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.224+0000 7f6d9b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:18:48.225 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.224+0000 7f6d9b7fe640 1 -- 172.21.15.6:0/2831991906 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f6d94001a10 con 0x7f6d9c0b1b00 2024-03-25T19:18:48.225 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.224+0000 7f6d9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:18:48.225 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.224+0000 7f6d9b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:18:48.225 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.224+0000 7f6d9b7fe640 10 monclient: dump: 2024-03-25T19:18:48.225 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:18:48.225 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:48.225 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:48.225 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:48.225 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:18:48.226 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:48.226 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:18:48.226 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:18:48.226 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:18:48.226 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:48.226 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.224+0000 7f6da2fac640 5 monclient: authenticate success, global_id 4435 2024-03-25T19:18:48.226 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.224+0000 7f6da2fac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:18:48.229 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.229+0000 7f6d9b7fe640 1 -- 172.21.15.6:0/2831991906 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d94001b70 con 0x7f6d9c0b1b00 2024-03-25T19:18:48.229 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.229+0000 7f6d9b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:18:48.229 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.229+0000 7f6da2fac640 10 monclient: get_monmap_and_config success 2024-03-25T19:18:48.229 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.229+0000 7f6da2fac640 4 set_mon_vals no callback set 2024-03-25T19:18:48.230 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.229+0000 7f6da2fac640 10 monclient: shutdown 2024-03-25T19:18:48.230 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.229+0000 7f6da2fac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:18:48.230 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.229+0000 7f6da2fac640 1 -- 172.21.15.6:0/2831991906 >> v1:172.21.15.192:6789/0 conn(0x7f6d9c0b1b00 legacy=0x7f6d9c14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:48.230 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.229+0000 7f6da2fac640 1 -- 172.21.15.6:0/2831991906 shutdown_connections 2024-03-25T19:18:48.230 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.229+0000 7f6da2fac640 1 -- 172.21.15.6:0/2831991906 >> 172.21.15.6:0/2831991906 conn(0x7f6d9c0617a0 msgr2=0x7f6d9c061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:18:48.230 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.229+0000 7f6da2fac640 1 -- 172.21.15.6:0/2831991906 shutdown_connections 2024-03-25T19:18:48.230 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.230+0000 7f6da2fac640 1 -- 172.21.15.6:0/2831991906 wait complete. 2024-03-25T19:18:48.232 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.230+0000 7f6da2fac640 10 monclient: build_initial_monmap 2024-03-25T19:18:48.232 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.230+0000 7f6da2fac640 10 monclient: monmap: 2024-03-25T19:18:48.232 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:18:48.232 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:18:48.232 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:18:48.232 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:18:48.232 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:18:48.232 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:18:48.232 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:18:48.232 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:18:48.232 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:18:48.232 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:48.232 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.231+0000 7f6da2fac640 1 librados: starting msgr at 2024-03-25T19:18:48.232 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.231+0000 7f6da2fac640 1 librados: starting objecter 2024-03-25T19:18:48.233 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.231+0000 7f6da2fac640 1 Processor -- start 2024-03-25T19:18:48.233 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.231+0000 7f6da2fac640 1 -- start start 2024-03-25T19:18:48.233 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.231+0000 7f6da2fac640 1 librados: setting wanted keys 2024-03-25T19:18:48.233 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.231+0000 7f6da2fac640 1 librados: calling monclient init 2024-03-25T19:18:48.233 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.231+0000 7f6da2fac640 10 monclient: init 2024-03-25T19:18:48.233 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.231+0000 7f6da2fac640 10 monclient: _reopen_session rank -1 2024-03-25T19:18:48.233 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.231+0000 7f6da2fac640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:18:48.233 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.231+0000 7f6da2fac640 10 monclient(hunting): picked mon.noname-b con 0x7f6d9c0b1b00 addr v1:172.21.15.192:6789/0 2024-03-25T19:18:48.233 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.231+0000 7f6da2fac640 10 monclient(hunting): picked mon.noname-c con 0x7f6d9c0b44b0 addr v1:172.21.15.6:6790/0 2024-03-25T19:18:48.233 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.231+0000 7f6da2fac640 10 monclient(hunting): picked mon.noname-a con 0x7f6d9c0b4e70 addr v1:172.21.15.6:6789/0 2024-03-25T19:18:48.233 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.231+0000 7f6da2fac640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9c135fb0 con 0x7f6d9c0b4e70 2024-03-25T19:18:48.233 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.231+0000 7f6da2fac640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9c136d10 con 0x7f6d9c0b1b00 2024-03-25T19:18:48.233 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.231+0000 7f6da2fac640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9c136ff0 con 0x7f6d9c0b44b0 2024-03-25T19:18:48.233 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.231+0000 7f6da2fac640 10 monclient(hunting): _renew_subs 2024-03-25T19:18:48.233 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.231+0000 7f6da2fac640 10 monclient(hunting): authenticate will time out at 6232.471705s 2024-03-25T19:18:48.248 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.248+0000 7f6d9bfff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f6d9c0b44b0 0x7f6d9c1417b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:32996/0 (socket says 172.21.15.6:32996) 2024-03-25T19:18:48.248 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.248+0000 7f6d9bfff640 1 -- 172.21.15.6:0/2165483304 learned_addr learned my addr 172.21.15.6:0/2165483304 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:18:48.253 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.253+0000 7f6d997fa640 10 client.?.objecter ms_handle_connect 0x7f6d9c0b44b0 2024-03-25T19:18:48.253 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.253+0000 7f6d997fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:18:48.259 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.259+0000 7f6d997fa640 10 client.?.objecter ms_handle_connect 0x7f6d9c0b4e70 2024-03-25T19:18:48.259 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.259+0000 7f6d997fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:18:48.260 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.260+0000 7f6d997fa640 10 client.?.objecter ms_handle_connect 0x7f6d9c0b1b00 2024-03-25T19:18:48.260 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.260+0000 7f6d997fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:18:48.270 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.270+0000 7f6d997fa640 1 -- 172.21.15.6:0/2165483304 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2862238830 0 0) 0x7f6d9c136ff0 con 0x7f6d9c0b44b0 2024-03-25T19:18:48.270 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.270+0000 7f6d997fa640 10 monclient(hunting): _negotiate 2024-03-25T19:18:48.271 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.270+0000 7f6d997fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:48.271 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.270+0000 7f6d997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:48.271 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.270+0000 7f6d997fa640 10 monclient(hunting): my global_id is 4436 2024-03-25T19:18:48.271 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.270+0000 7f6d997fa640 1 -- 172.21.15.6:0/2165483304 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d88004eb0 con 0x7f6d9c0b44b0 2024-03-25T19:18:48.285 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.284+0000 7f6d997fa640 1 -- 172.21.15.6:0/2165483304 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3215592704 0 0) 0x7f6d9c136d10 con 0x7f6d9c0b1b00 2024-03-25T19:18:48.285 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.285+0000 7f6d997fa640 10 monclient(hunting): _negotiate 2024-03-25T19:18:48.285 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.285+0000 7f6d997fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:48.285 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.285+0000 7f6d997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:48.285 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.285+0000 7f6d997fa640 10 monclient(hunting): my global_id is 4438 2024-03-25T19:18:48.285 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.285+0000 7f6d997fa640 1 -- 172.21.15.6:0/2165483304 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d9c136ff0 con 0x7f6d9c0b1b00 2024-03-25T19:18:48.297 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.296+0000 7f6d997fa640 1 -- 172.21.15.6:0/2165483304 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2948768724 0 0) 0x7f6d9c136ff0 con 0x7f6d9c0b1b00 2024-03-25T19:18:48.297 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.296+0000 7f6d997fa640 1 -- 172.21.15.6:0/2165483304 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d9c136d10 con 0x7f6d9c0b1b00 2024-03-25T19:18:48.297 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.296+0000 7f6d997fa640 1 -- 172.21.15.6:0/2165483304 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f6d8c001810 con 0x7f6d9c0b1b00 2024-03-25T19:18:48.297 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.296+0000 7f6d997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:18:48.297 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.296+0000 7f6d997fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:18:48.297 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.296+0000 7f6d997fa640 10 monclient(hunting): dump: 2024-03-25T19:18:48.297 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:18:48.297 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:48.297 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:48.297 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:48.297 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:18:48.297 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:48.298 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:18:48.298 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:18:48.298 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:18:48.298 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:48.298 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.296+0000 7f6d997fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:18:48.298 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.298+0000 7f6d997fa640 1 -- 172.21.15.6:0/2165483304 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3856277835 0 0) 0x7f6d88004eb0 con 0x7f6d9c0b44b0 2024-03-25T19:18:48.298 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.298+0000 7f6d997fa640 1 -- 172.21.15.6:0/2165483304 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d9c136ff0 con 0x7f6d9c0b44b0 2024-03-25T19:18:48.298 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.298+0000 7f6d997fa640 1 -- 172.21.15.6:0/2165483304 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f6d90001930 con 0x7f6d9c0b44b0 2024-03-25T19:18:48.299 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.298+0000 7f6d997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:18:48.299 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.298+0000 7f6d997fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:18:48.299 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.298+0000 7f6d997fa640 10 monclient(hunting): dump: 2024-03-25T19:18:48.299 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:18:48.299 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:48.299 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:48.299 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:48.299 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:18:48.299 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:48.299 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:18:48.299 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:18:48.299 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:18:48.299 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:48.303 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.303+0000 7f6d997fa640 1 -- 172.21.15.6:0/2165483304 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4116647469 0 0) 0x7f6d9c135fb0 con 0x7f6d9c0b4e70 2024-03-25T19:18:48.303 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.303+0000 7f6d997fa640 10 monclient(hunting): _negotiate 2024-03-25T19:18:48.303 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.303+0000 7f6d997fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:48.303 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.303+0000 7f6d997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:48.303 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.303+0000 7f6d997fa640 10 monclient(hunting): my global_id is 4434 2024-03-25T19:18:48.304 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.303+0000 7f6d997fa640 1 -- 172.21.15.6:0/2165483304 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d88004eb0 con 0x7f6d9c0b4e70 2024-03-25T19:18:48.332 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.331+0000 7f6d997fa640 1 -- 172.21.15.6:0/2165483304 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 799512447 0 0) 0x7f6d9c136d10 con 0x7f6d9c0b1b00 2024-03-25T19:18:48.332 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.331+0000 7f6d997fa640 1 -- 172.21.15.6:0/2165483304 >> v1:172.21.15.6:6790/0 conn(0x7f6d9c0b44b0 legacy=0x7f6d9c1417b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:48.332 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.331+0000 7f6d997fa640 1 -- 172.21.15.6:0/2165483304 >> v1:172.21.15.6:6789/0 conn(0x7f6d9c0b4e70 legacy=0x7f6d9c1e25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:48.332 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.331+0000 7f6d997fa640 10 monclient: _finish_hunting 0 2024-03-25T19:18:48.332 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.331+0000 7f6d997fa640 1 monclient: found mon.b 2024-03-25T19:18:48.332 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.331+0000 7f6d997fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:18:48.332 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.331+0000 7f6d997fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:18:48.332 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.331+0000 7f6d997fa640 1 -- 172.21.15.6:0/2165483304 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d9c1372d0 con 0x7f6d9c0b1b00 2024-03-25T19:18:48.332 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.331+0000 7f6d997fa640 10 monclient: _finish_auth 0 2024-03-25T19:18:48.332 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.331+0000 7f6d997fa640 10 monclient: _check_auth_tickets 2024-03-25T19:18:48.332 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.331+0000 7f6d997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:18:48.332 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.331+0000 7f6da2fac640 5 monclient: authenticate success, global_id 4438 2024-03-25T19:18:48.332 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.331+0000 7f6da2fac640 10 monclient: _renew_subs 2024-03-25T19:18:48.332 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.331+0000 7f6da2fac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:18:48.333 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.331+0000 7f6da2fac640 1 -- 172.21.15.6:0/2165483304 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d9c143140 con 0x7f6d9c0b1b00 2024-03-25T19:18:48.333 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.331+0000 7f6da2fac640 10 client.4438.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:18:48.333 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.331+0000 7f6da2fac640 10 monclient: _renew_subs 2024-03-25T19:18:48.333 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.331+0000 7f6da2fac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:18:48.333 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.331+0000 7f6da2fac640 1 -- 172.21.15.6:0/2165483304 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d9c135380 con 0x7f6d9c0b1b00 2024-03-25T19:18:48.333 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.331+0000 7f6da2fac640 1 librados: init done 2024-03-25T19:18:48.333 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.332+0000 7f6da2fac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:18:48.334 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.332+0000 7f6da2fac640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:18:48.334 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.332+0000 7f6da2fac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:18:48.334 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.332+0000 7f6da2fac640 1 -- 172.21.15.6:0/2165483304 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d9c04f730 con 0x7f6d9c0b1b00 2024-03-25T19:18:48.630 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.629+0000 7f6d997fa640 1 -- 172.21.15.6:0/2165483304 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d8c001970 con 0x7f6d9c0b1b00 2024-03-25T19:18:48.630 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.629+0000 7f6d997fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:18:48.630 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.629+0000 7f6d997fa640 1 -- 172.21.15.6:0/2165483304 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f6d8c0037b0 con 0x7f6d9c0b1b00 2024-03-25T19:18:48.630 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.629+0000 7f6d997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:18:48.630 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.629+0000 7f6d997fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:18:48.630 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.629+0000 7f6d997fa640 10 monclient: dump: 2024-03-25T19:18:48.630 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:18:48.630 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:48.630 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:48.630 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:48.630 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:18:48.631 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:48.631 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:18:48.631 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:18:48.631 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:18:48.631 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:48.631 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.629+0000 7f6d9affd640 4 set_mon_vals no callback set 2024-03-25T19:18:48.631 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.630+0000 7f6d997fa640 1 -- 172.21.15.6:0/2165483304 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f6d8c0039d0 con 0x7f6d9c0b1b00 2024-03-25T19:18:48.631 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.631+0000 7f6d9bfff640 10 monclient: get_auth_request con 0x7f6d8806df40 auth_method 0 2024-03-25T19:18:48.631 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.631+0000 7f6d997fa640 10 client.4438.objecter ms_handle_connect 0x7f6d8806df40 2024-03-25T19:18:48.635 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.635+0000 7f6d997fa640 1 -- 172.21.15.6:0/2165483304 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f6d8c053570 con 0x7f6d9c0b1b00 2024-03-25T19:18:48.635 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.635+0000 7f6d997fa640 10 client.4438.objecter ms_dispatch 0x7f6d9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:18:48.635 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.635+0000 7f6d997fa640 3 client.4438.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:18:48.635 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.635+0000 7f6d997fa640 3 client.4438.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:18:48.635 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.635+0000 7f6d997fa640 20 client.4438.objecter dump_active .. 0 homeless 2024-03-25T19:18:48.635 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.635+0000 7f6d997fa640 1 -- 172.21.15.6:0/2165483304 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6d8c04f080 con 0x7f6d9c0b1b00 2024-03-25T19:18:48.635 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.635+0000 7f6d997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:18:48.635 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.635+0000 7f6d997fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:18:48.882 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.881+0000 7f6da2fac640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:18:48.882 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.881+0000 7f6da2fac640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:18:48.882 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.881+0000 7f6da2fac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:18:48.882 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.881+0000 7f6da2fac640 1 -- 172.21.15.6:0/2165483304 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d9c13f360 con 0x7f6d9c0b1b00 2024-03-25T19:18:48.920 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.920+0000 7f6d997fa640 1 -- 172.21.15.6:0/2165483304 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f6d8c050fb0 con 0x7f6d9c0b1b00 2024-03-25T19:18:48.920 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.920+0000 7f6d997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:18:48.920 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.920+0000 7f6d997fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:18:48.920 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:18:48.921 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:18:48.923 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.923+0000 7f6da2fac640 10 librados: watch_flush enter 2024-03-25T19:18:48.923 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.923+0000 7f6da2fac640 10 librados: watch_flush exit 2024-03-25T19:18:48.923 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.923+0000 7f6da2fac640 20 client.4438.objecter shutdown clearing up homeless session... 2024-03-25T19:18:48.924 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.923+0000 7f6da2fac640 10 client.4438.objecter successfully canceled tick 2024-03-25T19:18:48.924 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.923+0000 7f6da2fac640 1 -- 172.21.15.6:0/2165483304 >> v1:172.21.15.6:6800/2936652650 conn(0x7f6d8806df40 legacy=0x7f6d88070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:48.924 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.923+0000 7f6da2fac640 10 monclient: shutdown 2024-03-25T19:18:48.924 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.923+0000 7f6da2fac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:18:48.924 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.923+0000 7f6da2fac640 1 -- 172.21.15.6:0/2165483304 >> v1:172.21.15.192:6789/0 conn(0x7f6d9c0b1b00 legacy=0x7f6d9c1376f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:48.924 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.923+0000 7f6da2fac640 1 -- 172.21.15.6:0/2165483304 shutdown_connections 2024-03-25T19:18:48.924 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.923+0000 7f6da2fac640 1 -- 172.21.15.6:0/2165483304 >> 172.21.15.6:0/2165483304 conn(0x7f6d9c0617a0 msgr2=0x7f6d9c0b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:18:48.924 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.923+0000 7f6da2fac640 1 -- 172.21.15.6:0/2165483304 shutdown_connections 2024-03-25T19:18:48.924 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.923+0000 7f6da2fac640 1 -- 172.21.15.6:0/2165483304 wait complete. 2024-03-25T19:18:48.924 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:48.924+0000 7f6da2fac640 1 librados: shutdown 2024-03-25T19:18:48.936 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:18:55.743 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:18:55.861 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.860+0000 7fe999356640 10 monclient: get_monmap_and_config 2024-03-25T19:18:55.861 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.860+0000 7fe999356640 10 monclient: build_initial_monmap 2024-03-25T19:18:55.861 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.860+0000 7fe999356640 10 monclient: monmap: 2024-03-25T19:18:55.861 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:18:55.861 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:55.861 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:18:55.861838+0000 2024-03-25T19:18:55.861 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:18:55.861838+0000 2024-03-25T19:18:55.861 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:18:55.861 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:55.861 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:18:55.862 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:18:55.862 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:18:55.862 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:55.863 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.861+0000 7fe999356640 1 Processor -- start 2024-03-25T19:18:55.863 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.861+0000 7fe999356640 1 -- start start 2024-03-25T19:18:55.863 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.861+0000 7fe999356640 10 monclient: init 2024-03-25T19:18:55.863 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.862+0000 7fe999356640 10 monclient: _reopen_session rank -1 2024-03-25T19:18:55.863 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.862+0000 7fe999356640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:18:55.863 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.862+0000 7fe999356640 10 monclient(hunting): picked mon.noname-a con 0x7fe9940b4ff0 addr v1:172.21.15.6:6789/0 2024-03-25T19:18:55.863 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.862+0000 7fe999356640 10 monclient(hunting): picked mon.noname-c con 0x7fe9940b4670 addr v1:172.21.15.192:6789/0 2024-03-25T19:18:55.864 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.862+0000 7fe999356640 10 monclient(hunting): picked mon.noname-b con 0x7fe99413c480 addr v1:172.21.15.6:6790/0 2024-03-25T19:18:55.864 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.862+0000 7fe999356640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9940ab0e0 con 0x7fe9940b4ff0 2024-03-25T19:18:55.864 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.862+0000 7fe999356640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9940ab3c0 con 0x7fe9940b4670 2024-03-25T19:18:55.864 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.862+0000 7fe999356640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9940a9f30 con 0x7fe99413c480 2024-03-25T19:18:55.864 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.862+0000 7fe999356640 10 monclient(hunting): _renew_subs 2024-03-25T19:18:55.864 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.862+0000 7fe999356640 10 monclient(hunting): authenticate will time out at 6240.102746s 2024-03-25T19:18:55.872 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.872+0000 7fe9937fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fe99413c480 0x7fe99414dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46896/0 (socket says 172.21.15.6:46896) 2024-03-25T19:18:55.872 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.872+0000 7fe9937fe640 1 -- 172.21.15.6:0/3132701370 learned_addr learned my addr 172.21.15.6:0/3132701370 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:18:55.887 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.886+0000 7fe9927fc640 1 -- 172.21.15.6:0/3132701370 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4029120303 0 0) 0x7fe9940a9f30 con 0x7fe99413c480 2024-03-25T19:18:55.887 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.887+0000 7fe9927fc640 10 monclient(hunting): _negotiate 2024-03-25T19:18:55.887 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.887+0000 7fe9927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:55.887 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.887+0000 7fe9927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:55.887 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.887+0000 7fe9927fc640 10 monclient(hunting): my global_id is 4439 2024-03-25T19:18:55.887 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.887+0000 7fe9927fc640 1 -- 172.21.15.6:0/3132701370 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe980003520 con 0x7fe99413c480 2024-03-25T19:18:55.889 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.890+0000 7fe9927fc640 1 -- 172.21.15.6:0/3132701370 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1150300798 0 0) 0x7fe9940ab0e0 con 0x7fe9940b4ff0 2024-03-25T19:18:55.890 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.890+0000 7fe9927fc640 10 monclient(hunting): _negotiate 2024-03-25T19:18:55.890 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.890+0000 7fe9927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:55.890 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.890+0000 7fe9927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:55.890 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.890+0000 7fe9927fc640 10 monclient(hunting): my global_id is 4437 2024-03-25T19:18:55.890 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.890+0000 7fe9927fc640 1 -- 172.21.15.6:0/3132701370 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9940a9f30 con 0x7fe9940b4ff0 2024-03-25T19:18:55.896 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.896+0000 7fe9927fc640 1 -- 172.21.15.6:0/3132701370 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1481737458 0 0) 0x7fe9940ab3c0 con 0x7fe9940b4670 2024-03-25T19:18:55.896 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.896+0000 7fe9927fc640 10 monclient(hunting): _negotiate 2024-03-25T19:18:55.896 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.896+0000 7fe9927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:55.896 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.896+0000 7fe9927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:55.896 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.896+0000 7fe9927fc640 10 monclient(hunting): my global_id is 4441 2024-03-25T19:18:55.896 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.896+0000 7fe9927fc640 1 -- 172.21.15.6:0/3132701370 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9940ab0e0 con 0x7fe9940b4670 2024-03-25T19:18:55.900 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.899+0000 7fe9927fc640 1 -- 172.21.15.6:0/3132701370 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3479720764 0 0) 0x7fe9940a9f30 con 0x7fe9940b4ff0 2024-03-25T19:18:55.900 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.899+0000 7fe9927fc640 1 -- 172.21.15.6:0/3132701370 >> v1:172.21.15.6:6790/0 conn(0x7fe99413c480 legacy=0x7fe99414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:55.900 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.900+0000 7fe9927fc640 1 -- 172.21.15.6:0/3132701370 >> v1:172.21.15.192:6789/0 conn(0x7fe9940b4670 legacy=0x7fe9941425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:55.900 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.900+0000 7fe9927fc640 10 monclient: _finish_hunting 0 2024-03-25T19:18:55.900 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.900+0000 7fe9927fc640 1 monclient: found mon.noname-a 2024-03-25T19:18:55.900 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.900+0000 7fe9927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:18:55.900 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.900+0000 7fe9927fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-03-25T19:18:55.900 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.900+0000 7fe9927fc640 1 -- 172.21.15.6:0/3132701370 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9940a9b70 con 0x7fe9940b4ff0 2024-03-25T19:18:55.901 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.900+0000 7fe9927fc640 10 monclient: _finish_auth 0 2024-03-25T19:18:55.901 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.900+0000 7fe9927fc640 10 monclient: _check_auth_tickets 2024-03-25T19:18:55.901 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.900+0000 7fe9927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:18:55.901 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.900+0000 7fe9927fc640 1 -- 172.21.15.6:0/3132701370 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fe984001670 con 0x7fe9940b4ff0 2024-03-25T19:18:55.901 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.900+0000 7fe9927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:18:55.901 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.900+0000 7fe9927fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:18:55.901 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.900+0000 7fe9927fc640 10 monclient: dump: 2024-03-25T19:18:55.901 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:18:55.901 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:55.901 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:55.901 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:55.901 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:18:55.901 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:55.901 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:18:55.902 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:18:55.902 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:18:55.902 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:55.902 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.900+0000 7fe999356640 5 monclient: authenticate success, global_id 4437 2024-03-25T19:18:55.902 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.900+0000 7fe999356640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:18:55.912 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.912+0000 7fe9927fc640 1 -- 172.21.15.6:0/3132701370 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9840017d0 con 0x7fe9940b4ff0 2024-03-25T19:18:55.912 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.912+0000 7fe9927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:18:55.912 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.912+0000 7fe9927fc640 1 -- 172.21.15.6:0/3132701370 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fe984001be0 con 0x7fe9940b4ff0 2024-03-25T19:18:55.912 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.912+0000 7fe999356640 10 monclient: get_monmap_and_config success 2024-03-25T19:18:55.912 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.912+0000 7fe999356640 4 set_mon_vals no callback set 2024-03-25T19:18:55.912 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.912+0000 7fe9927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:18:55.913 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.912+0000 7fe9927fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:18:55.913 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.912+0000 7fe9927fc640 10 monclient: dump: 2024-03-25T19:18:55.913 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:18:55.913 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:55.913 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:55.913 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:55.913 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:18:55.913 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:55.913 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:18:55.913 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:18:55.913 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:18:55.913 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:55.915 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.913+0000 7fe999356640 10 monclient: shutdown 2024-03-25T19:18:55.916 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.913+0000 7fe999356640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:18:55.916 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.913+0000 7fe999356640 1 -- 172.21.15.6:0/3132701370 >> v1:172.21.15.6:6789/0 conn(0x7fe9940b4ff0 legacy=0x7fe99413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:55.916 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.913+0000 7fe999356640 1 -- 172.21.15.6:0/3132701370 shutdown_connections 2024-03-25T19:18:55.916 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.913+0000 7fe999356640 1 -- 172.21.15.6:0/3132701370 >> 172.21.15.6:0/3132701370 conn(0x7fe9940617a0 msgr2=0x7fe994061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:18:55.916 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.913+0000 7fe999356640 1 -- 172.21.15.6:0/3132701370 shutdown_connections 2024-03-25T19:18:55.916 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.913+0000 7fe999356640 1 -- 172.21.15.6:0/3132701370 wait complete. 2024-03-25T19:18:55.916 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.914+0000 7fe999356640 10 monclient: build_initial_monmap 2024-03-25T19:18:55.916 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.914+0000 7fe999356640 10 monclient: monmap: 2024-03-25T19:18:55.916 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:18:55.916 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:18:55.916 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:18:55.916 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:18:55.916 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:18:55.917 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:18:55.917 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:18:55.917 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:18:55.917 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:18:55.917 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:55.917 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.914+0000 7fe999356640 1 librados: starting msgr at 2024-03-25T19:18:55.917 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.914+0000 7fe999356640 1 librados: starting objecter 2024-03-25T19:18:55.917 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.914+0000 7fe999356640 1 Processor -- start 2024-03-25T19:18:55.917 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.914+0000 7fe999356640 1 -- start start 2024-03-25T19:18:55.917 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.914+0000 7fe999356640 1 librados: setting wanted keys 2024-03-25T19:18:55.917 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.914+0000 7fe999356640 1 librados: calling monclient init 2024-03-25T19:18:55.917 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.914+0000 7fe999356640 10 monclient: init 2024-03-25T19:18:55.917 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.915+0000 7fe999356640 10 monclient: _reopen_session rank -1 2024-03-25T19:18:55.917 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.915+0000 7fe999356640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:18:55.918 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.915+0000 7fe999356640 10 monclient(hunting): picked mon.noname-a con 0x7fe9940b4670 addr v1:172.21.15.6:6789/0 2024-03-25T19:18:55.918 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.915+0000 7fe999356640 10 monclient(hunting): picked mon.noname-b con 0x7fe9940b4ff0 addr v1:172.21.15.192:6789/0 2024-03-25T19:18:55.918 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.915+0000 7fe999356640 10 monclient(hunting): picked mon.noname-c con 0x7fe99413c480 addr v1:172.21.15.6:6790/0 2024-03-25T19:18:55.918 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.915+0000 7fe999356640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe994130cc0 con 0x7fe9940b4670 2024-03-25T19:18:55.918 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.915+0000 7fe999356640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9941445e0 con 0x7fe9940b4ff0 2024-03-25T19:18:55.918 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.915+0000 7fe999356640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9941448c0 con 0x7fe99413c480 2024-03-25T19:18:55.918 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.915+0000 7fe999356640 10 monclient(hunting): _renew_subs 2024-03-25T19:18:55.918 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.915+0000 7fe999356640 10 monclient(hunting): authenticate will time out at 6240.155072s 2024-03-25T19:18:55.926 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.926+0000 7fe98bfff640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fe9940b4ff0 0x7fe994145070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:58660/0 (socket says 172.21.15.6:58660) 2024-03-25T19:18:55.926 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.926+0000 7fe98bfff640 1 -- 172.21.15.6:0/1568596672 learned_addr learned my addr 172.21.15.6:0/1568596672 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:18:55.926 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.926+0000 7fe98b7fe640 10 client.?.objecter ms_handle_connect 0x7fe9940b4ff0 2024-03-25T19:18:55.926 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.926+0000 7fe98b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:18:55.933 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.933+0000 7fe98b7fe640 10 client.?.objecter ms_handle_connect 0x7fe99413c480 2024-03-25T19:18:55.933 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.933+0000 7fe98b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:18:55.936 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.936+0000 7fe98b7fe640 1 -- 172.21.15.6:0/1568596672 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2131337806 0 0) 0x7fe9941445e0 con 0x7fe9940b4ff0 2024-03-25T19:18:55.936 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.936+0000 7fe98b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:18:55.936 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.936+0000 7fe98b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:55.936 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.936+0000 7fe98b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:55.936 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.936+0000 7fe98b7fe640 10 monclient(hunting): my global_id is 4444 2024-03-25T19:18:55.937 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.936+0000 7fe98b7fe640 1 -- 172.21.15.6:0/1568596672 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe974004eb0 con 0x7fe9940b4ff0 2024-03-25T19:18:55.941 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.941+0000 7fe98b7fe640 10 client.?.objecter ms_handle_connect 0x7fe9940b4670 2024-03-25T19:18:55.941 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.941+0000 7fe98b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:18:55.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.958+0000 7fe98b7fe640 1 -- 172.21.15.6:0/1568596672 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2117297218 0 0) 0x7fe9941448c0 con 0x7fe99413c480 2024-03-25T19:18:55.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.958+0000 7fe98b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:18:55.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.958+0000 7fe98b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:55.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.958+0000 7fe98b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:55.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.958+0000 7fe98b7fe640 10 monclient(hunting): my global_id is 4442 2024-03-25T19:18:55.959 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.958+0000 7fe98b7fe640 1 -- 172.21.15.6:0/1568596672 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9941445e0 con 0x7fe99413c480 2024-03-25T19:18:55.964 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.964+0000 7fe98b7fe640 1 -- 172.21.15.6:0/1568596672 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 919835563 0 0) 0x7fe994130cc0 con 0x7fe9940b4670 2024-03-25T19:18:55.964 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.964+0000 7fe98b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:18:55.964 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.964+0000 7fe98b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:18:55.964 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.964+0000 7fe98b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:18:55.964 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.964+0000 7fe98b7fe640 10 monclient(hunting): my global_id is 4440 2024-03-25T19:18:55.964 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.964+0000 7fe98b7fe640 1 -- 172.21.15.6:0/1568596672 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9941448c0 con 0x7fe9940b4670 2024-03-25T19:18:55.970 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.970+0000 7fe98b7fe640 1 -- 172.21.15.6:0/1568596672 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2936358328 0 0) 0x7fe974004eb0 con 0x7fe9940b4ff0 2024-03-25T19:18:55.970 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.970+0000 7fe98b7fe640 1 -- 172.21.15.6:0/1568596672 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe994130cc0 con 0x7fe9940b4ff0 2024-03-25T19:18:55.970 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.970+0000 7fe98b7fe640 1 -- 172.21.15.6:0/1568596672 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fe97c001a10 con 0x7fe9940b4ff0 2024-03-25T19:18:55.970 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.970+0000 7fe98b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:18:55.970 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.970+0000 7fe98b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:18:55.970 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.970+0000 7fe98b7fe640 10 monclient(hunting): dump: 2024-03-25T19:18:55.970 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:18:55.970 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:55.970 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:55.971 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:55.971 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:18:55.971 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:55.971 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:18:55.971 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:18:55.971 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:18:55.971 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:55.971 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.970+0000 7fe98b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:18:55.991 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.989+0000 7fe98b7fe640 1 -- 172.21.15.6:0/1568596672 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2496219575 0 0) 0x7fe9941448c0 con 0x7fe9940b4670 2024-03-25T19:18:55.991 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.989+0000 7fe98b7fe640 1 -- 172.21.15.6:0/1568596672 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe974004eb0 con 0x7fe9940b4670 2024-03-25T19:18:55.991 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.989+0000 7fe98b7fe640 1 -- 172.21.15.6:0/1568596672 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fe984001ae0 con 0x7fe9940b4670 2024-03-25T19:18:55.991 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.989+0000 7fe98b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:18:55.991 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.989+0000 7fe98b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:18:55.991 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.989+0000 7fe98b7fe640 10 monclient(hunting): dump: 2024-03-25T19:18:55.991 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:18:55.991 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:55.991 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:55.991 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:55.991 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:18:55.991 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:55.991 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:18:55.991 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:18:55.992 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:18:55.992 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:55.992 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.989+0000 7fe98b7fe640 1 -- 172.21.15.6:0/1568596672 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 608693330 0 0) 0x7fe994130cc0 con 0x7fe9940b4ff0 2024-03-25T19:18:55.992 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.989+0000 7fe98b7fe640 1 -- 172.21.15.6:0/1568596672 >> v1:172.21.15.6:6790/0 conn(0x7fe99413c480 legacy=0x7fe994145780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:55.992 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.989+0000 7fe98b7fe640 1 -- 172.21.15.6:0/1568596672 >> v1:172.21.15.6:6789/0 conn(0x7fe9940b4670 legacy=0x7fe99413cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:55.992 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.989+0000 7fe98b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:18:55.992 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.989+0000 7fe98b7fe640 1 monclient: found mon.b 2024-03-25T19:18:55.992 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.989+0000 7fe98b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:18:55.992 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.989+0000 7fe98b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:18:55.992 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.989+0000 7fe98b7fe640 1 -- 172.21.15.6:0/1568596672 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe994146530 con 0x7fe9940b4ff0 2024-03-25T19:18:55.992 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.989+0000 7fe98b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:18:55.992 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.989+0000 7fe98b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:18:55.992 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.989+0000 7fe98b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:18:55.992 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.989+0000 7fe999356640 5 monclient: authenticate success, global_id 4444 2024-03-25T19:18:55.992 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.989+0000 7fe999356640 10 monclient: _renew_subs 2024-03-25T19:18:55.992 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.989+0000 7fe999356640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:18:55.993 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.989+0000 7fe999356640 1 -- 172.21.15.6:0/1568596672 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe99413ee20 con 0x7fe9940b4ff0 2024-03-25T19:18:55.993 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.989+0000 7fe999356640 10 client.4444.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:18:55.993 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.989+0000 7fe999356640 10 monclient: _renew_subs 2024-03-25T19:18:55.993 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.989+0000 7fe999356640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:18:55.993 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.989+0000 7fe999356640 1 -- 172.21.15.6:0/1568596672 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe99413d6e0 con 0x7fe9940b4ff0 2024-03-25T19:18:55.993 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.989+0000 7fe999356640 1 librados: init done 2024-03-25T19:18:55.993 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.991+0000 7fe999356640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:18:55.993 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.991+0000 7fe999356640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:18:55.993 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.991+0000 7fe999356640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:18:55.993 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:55.991+0000 7fe999356640 1 -- 172.21.15.6:0/1568596672 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe99406dbd0 con 0x7fe9940b4ff0 2024-03-25T19:18:56.206 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:56.206+0000 7fe98b7fe640 1 -- 172.21.15.6:0/1568596672 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe97c001c70 con 0x7fe9940b4ff0 2024-03-25T19:18:56.206 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:56.206+0000 7fe98b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:18:56.206 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:56.206+0000 7fe991ffb640 4 set_mon_vals no callback set 2024-03-25T19:18:56.265 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:56.265+0000 7fe98b7fe640 1 -- 172.21.15.6:0/1568596672 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fe97c003a60 con 0x7fe9940b4ff0 2024-03-25T19:18:56.266 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:56.266+0000 7fe98b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:18:56.266 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:56.266+0000 7fe98b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:18:56.266 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:56.266+0000 7fe98b7fe640 10 monclient: dump: 2024-03-25T19:18:56.266 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:18:56.266 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:18:56.266 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:56.266 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:18:56.266 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:18:56.266 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:18:56.266 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:18:56.266 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:18:56.266 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:18:56.266 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:18:56.267 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:56.266+0000 7fe98b7fe640 1 -- 172.21.15.6:0/1568596672 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7fe97c003c40 con 0x7fe9940b4ff0 2024-03-25T19:18:56.267 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:56.267+0000 7fe992ffd640 10 monclient: get_auth_request con 0x7fe97406df70 auth_method 0 2024-03-25T19:18:56.267 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:56.267+0000 7fe98b7fe640 10 client.4444.objecter ms_handle_connect 0x7fe97406df70 2024-03-25T19:18:56.269 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:56.269+0000 7fe98b7fe640 1 -- 172.21.15.6:0/1568596672 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7fe97c0548f0 con 0x7fe9940b4ff0 2024-03-25T19:18:56.269 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:56.269+0000 7fe98b7fe640 10 client.4444.objecter ms_dispatch 0x7fe994000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:18:56.269 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:56.269+0000 7fe98b7fe640 3 client.4444.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:18:56.269 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:56.269+0000 7fe98b7fe640 3 client.4444.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:18:56.269 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:56.269+0000 7fe98b7fe640 20 client.4444.objecter dump_active .. 0 homeless 2024-03-25T19:18:56.273 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:56.273+0000 7fe98b7fe640 1 -- 172.21.15.6:0/1568596672 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe97c0b18b0 con 0x7fe9940b4ff0 2024-03-25T19:18:56.273 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:56.273+0000 7fe98b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:18:56.273 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:56.273+0000 7fe98b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:18:56.488 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:56.488+0000 7fe999356640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:18:56.488 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:56.488+0000 7fe999356640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:18:56.489 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:56.488+0000 7fe999356640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:18:56.489 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:56.488+0000 7fe999356640 1 -- 172.21.15.6:0/1568596672 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9941466c0 con 0x7fe9940b4ff0 2024-03-25T19:18:56.601 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:56.601+0000 7fe98b7fe640 1 -- 172.21.15.6:0/1568596672 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7fe97c050230 con 0x7fe9940b4ff0 2024-03-25T19:18:56.602 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:56.601+0000 7fe98b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:18:56.602 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:56.601+0000 7fe98b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:18:56.602 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:18:56.602 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:18:56.605 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:56.604+0000 7fe999356640 10 librados: watch_flush enter 2024-03-25T19:18:56.605 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:56.604+0000 7fe999356640 10 librados: watch_flush exit 2024-03-25T19:18:56.605 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:56.604+0000 7fe999356640 20 client.4444.objecter shutdown clearing up homeless session... 2024-03-25T19:18:56.605 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:56.604+0000 7fe999356640 10 client.4444.objecter successfully canceled tick 2024-03-25T19:18:56.605 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:56.604+0000 7fe999356640 1 -- 172.21.15.6:0/1568596672 >> v1:172.21.15.6:6800/2936652650 conn(0x7fe97406df70 legacy=0x7fe974070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:56.605 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:56.604+0000 7fe999356640 10 monclient: shutdown 2024-03-25T19:18:56.605 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:56.604+0000 7fe999356640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:18:56.605 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:56.604+0000 7fe999356640 1 -- 172.21.15.6:0/1568596672 >> v1:172.21.15.192:6789/0 conn(0x7fe9940b4ff0 legacy=0x7fe994145070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:18:56.605 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:56.604+0000 7fe999356640 1 -- 172.21.15.6:0/1568596672 shutdown_connections 2024-03-25T19:18:56.605 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:56.604+0000 7fe999356640 1 -- 172.21.15.6:0/1568596672 >> 172.21.15.6:0/1568596672 conn(0x7fe9940617a0 msgr2=0x7fe9941405c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:18:56.606 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:56.605+0000 7fe999356640 1 -- 172.21.15.6:0/1568596672 shutdown_connections 2024-03-25T19:18:56.606 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:56.605+0000 7fe999356640 1 -- 172.21.15.6:0/1568596672 wait complete. 2024-03-25T19:18:56.606 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:18:56.605+0000 7fe999356640 1 librados: shutdown 2024-03-25T19:18:56.617 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:19:03.341 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:05:20.688 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:19:03.350 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:05:20.697 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:19:03.393 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:05:20.740 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:19:03.402 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:05:20.720 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:19:03.404 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:05:20.751 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:19:03.408 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:05:20.726 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:19:03.420 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:05:20.767 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:19:03.422 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:19:03.436 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:05:20.754 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:19:03.456 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:05:20.802 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:19:03.529 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.529+0000 7f0c3f6f6640 10 monclient: get_monmap_and_config 2024-03-25T19:19:03.529 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.529+0000 7f0c3f6f6640 10 monclient: build_initial_monmap 2024-03-25T19:19:03.529 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.529+0000 7f0c3f6f6640 10 monclient: monmap: 2024-03-25T19:19:03.529 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:19:03.529 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:03.529 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:19:03.530436+0000 2024-03-25T19:19:03.529 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:19:03.530436+0000 2024-03-25T19:19:03.530 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:03.530 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:03.530 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:19:03.530 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:19:03.530 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:19:03.530 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:03.531 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.530+0000 7f0c3f6f6640 1 Processor -- start 2024-03-25T19:19:03.531 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.530+0000 7f0c3f6f6640 1 -- start start 2024-03-25T19:19:03.531 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.530+0000 7f0c3f6f6640 10 monclient: init 2024-03-25T19:19:03.531 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.530+0000 7f0c3f6f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:03.531 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.530+0000 7f0c3f6f6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:19:03.532 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.530+0000 7f0c3f6f6640 10 monclient(hunting): picked mon.noname-b con 0x7f0c380b4ff0 addr v1:172.21.15.6:6790/0 2024-03-25T19:19:03.532 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.531+0000 7f0c3f6f6640 10 monclient(hunting): picked mon.noname-c con 0x7f0c380b4670 addr v1:172.21.15.192:6789/0 2024-03-25T19:19:03.532 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.531+0000 7f0c3f6f6640 10 monclient(hunting): picked mon.noname-a con 0x7f0c3813c480 addr v1:172.21.15.6:6789/0 2024-03-25T19:19:03.532 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.531+0000 7f0c3f6f6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c380ab0e0 con 0x7f0c3813c480 2024-03-25T19:19:03.532 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.531+0000 7f0c3f6f6640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c380ab3c0 con 0x7f0c380b4670 2024-03-25T19:19:03.532 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.531+0000 7f0c3f6f6640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c380a9f30 con 0x7f0c380b4ff0 2024-03-25T19:19:03.532 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.531+0000 7f0c3f6f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:03.532 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.531+0000 7f0c3f6f6640 10 monclient(hunting): authenticate will time out at 6247.770860s 2024-03-25T19:19:03.537 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.538+0000 7f0c3dc93640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0c3813c480 0x7f0c3814dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50484/0 (socket says 172.21.15.6:50484) 2024-03-25T19:19:03.538 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.538+0000 7f0c3dc93640 1 -- 172.21.15.6:0/4171651907 learned_addr learned my addr 172.21.15.6:0/4171651907 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:19:03.553 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.553+0000 7f0c2ffff640 1 -- 172.21.15.6:0/4171651907 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2054357655 0 0) 0x7f0c380ab0e0 con 0x7f0c3813c480 2024-03-25T19:19:03.554 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.553+0000 7f0c2ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:19:03.554 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.553+0000 7f0c2ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:03.554 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.553+0000 7f0c2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:03.554 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.553+0000 7f0c2ffff640 10 monclient(hunting): my global_id is 4443 2024-03-25T19:19:03.554 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.554+0000 7f0c2ffff640 1 -- 172.21.15.6:0/4171651907 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c24003520 con 0x7f0c3813c480 2024-03-25T19:19:03.556 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.556+0000 7f0c2ffff640 1 -- 172.21.15.6:0/4171651907 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 742741464 0 0) 0x7f0c380a9f30 con 0x7f0c380b4ff0 2024-03-25T19:19:03.556 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.556+0000 7f0c2ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:19:03.556 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.556+0000 7f0c2ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:03.556 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.556+0000 7f0c2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:03.556 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.556+0000 7f0c2ffff640 10 monclient(hunting): my global_id is 4445 2024-03-25T19:19:03.556 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.556+0000 7f0c2ffff640 1 -- 172.21.15.6:0/4171651907 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c380ab0e0 con 0x7f0c380b4ff0 2024-03-25T19:19:03.568 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.567+0000 7f0c2ffff640 1 -- 172.21.15.6:0/4171651907 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2846881956 0 0) 0x7f0c24003520 con 0x7f0c3813c480 2024-03-25T19:19:03.568 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.568+0000 7f0c2ffff640 1 -- 172.21.15.6:0/4171651907 >> v1:172.21.15.6:6790/0 conn(0x7f0c380b4ff0 legacy=0x7f0c3813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:03.568 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.568+0000 7f0c2ffff640 1 -- 172.21.15.6:0/4171651907 >> v1:172.21.15.192:6789/0 conn(0x7f0c380b4670 legacy=0x7f0c381425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:03.568 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.568+0000 7f0c2ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:19:03.568 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.568+0000 7f0c2ffff640 1 monclient: found mon.noname-a 2024-03-25T19:19:03.568 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.568+0000 7f0c2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:19:03.568 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.568+0000 7f0c2ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-03-25T19:19:03.568 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.568+0000 7f0c2ffff640 1 -- 172.21.15.6:0/4171651907 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c380a9b70 con 0x7f0c3813c480 2024-03-25T19:19:03.568 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.568+0000 7f0c2ffff640 10 monclient: _finish_auth 0 2024-03-25T19:19:03.568 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.568+0000 7f0c2ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:19:03.568 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.568+0000 7f0c2ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:19:03.569 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.568+0000 7f0c2ffff640 1 -- 172.21.15.6:0/4171651907 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f0c30001a10 con 0x7f0c3813c480 2024-03-25T19:19:03.569 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.568+0000 7f0c2ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:03.569 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.568+0000 7f0c2ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:19:03.569 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.568+0000 7f0c2ffff640 10 monclient: dump: 2024-03-25T19:19:03.569 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:19:03.569 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:03.569 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:03.569 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:03.569 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:03.569 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:03.569 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:19:03.569 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:19:03.569 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:19:03.569 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:03.569 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.568+0000 7f0c3f6f6640 5 monclient: authenticate success, global_id 4443 2024-03-25T19:19:03.569 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.568+0000 7f0c3f6f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:19:03.578 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.578+0000 7f0c2ffff640 1 -- 172.21.15.6:0/4171651907 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c30001b70 con 0x7f0c3813c480 2024-03-25T19:19:03.578 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.578+0000 7f0c2ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:19:03.578 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.578+0000 7f0c2ffff640 1 -- 172.21.15.6:0/4171651907 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f0c30001f80 con 0x7f0c3813c480 2024-03-25T19:19:03.578 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.578+0000 7f0c2ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:03.578 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.578+0000 7f0c2ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:19:03.578 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.578+0000 7f0c2ffff640 10 monclient: dump: 2024-03-25T19:19:03.578 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:19:03.578 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:03.578 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:03.578 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:03.579 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:03.579 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:03.579 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:19:03.579 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:19:03.579 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:19:03.579 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:03.579 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.578+0000 7f0c3f6f6640 10 monclient: get_monmap_and_config success 2024-03-25T19:19:03.579 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.578+0000 7f0c3f6f6640 4 set_mon_vals no callback set 2024-03-25T19:19:03.580 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.579+0000 7f0c3f6f6640 10 monclient: shutdown 2024-03-25T19:19:03.581 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.579+0000 7f0c3f6f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:19:03.581 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.579+0000 7f0c3f6f6640 1 -- 172.21.15.6:0/4171651907 >> v1:172.21.15.6:6789/0 conn(0x7f0c3813c480 legacy=0x7f0c3814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:03.581 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.579+0000 7f0c3f6f6640 1 -- 172.21.15.6:0/4171651907 shutdown_connections 2024-03-25T19:19:03.581 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.579+0000 7f0c3f6f6640 1 -- 172.21.15.6:0/4171651907 >> 172.21.15.6:0/4171651907 conn(0x7f0c380617a0 msgr2=0x7f0c38061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:19:03.581 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.579+0000 7f0c3f6f6640 1 -- 172.21.15.6:0/4171651907 shutdown_connections 2024-03-25T19:19:03.581 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.579+0000 7f0c3f6f6640 1 -- 172.21.15.6:0/4171651907 wait complete. 2024-03-25T19:19:03.581 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.580+0000 7f0c3f6f6640 10 monclient: build_initial_monmap 2024-03-25T19:19:03.581 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.580+0000 7f0c3f6f6640 10 monclient: monmap: 2024-03-25T19:19:03.581 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:19:03.581 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:19:03.581 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:19:03.581 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:19:03.581 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:03.582 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:19:03.582 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:19:03.582 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:19:03.582 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:19:03.582 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:03.582 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.580+0000 7f0c3f6f6640 1 librados: starting msgr at 2024-03-25T19:19:03.582 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.580+0000 7f0c3f6f6640 1 librados: starting objecter 2024-03-25T19:19:03.582 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.580+0000 7f0c3f6f6640 1 Processor -- start 2024-03-25T19:19:03.582 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.580+0000 7f0c3f6f6640 1 -- start start 2024-03-25T19:19:03.582 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.580+0000 7f0c3f6f6640 1 librados: setting wanted keys 2024-03-25T19:19:03.582 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.580+0000 7f0c3f6f6640 1 librados: calling monclient init 2024-03-25T19:19:03.582 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.580+0000 7f0c3f6f6640 10 monclient: init 2024-03-25T19:19:03.584 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.581+0000 7f0c3f6f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:03.584 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.581+0000 7f0c3f6f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:19:03.584 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.581+0000 7f0c3f6f6640 10 monclient(hunting): picked mon.noname-b con 0x7f0c380b4670 addr v1:172.21.15.192:6789/0 2024-03-25T19:19:03.584 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.581+0000 7f0c3f6f6640 10 monclient(hunting): picked mon.noname-a con 0x7f0c380b4ff0 addr v1:172.21.15.6:6789/0 2024-03-25T19:19:03.584 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.581+0000 7f0c3f6f6640 10 monclient(hunting): picked mon.noname-c con 0x7f0c3813c480 addr v1:172.21.15.6:6790/0 2024-03-25T19:19:03.584 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.581+0000 7f0c3f6f6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c380acfd0 con 0x7f0c380b4ff0 2024-03-25T19:19:03.584 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.581+0000 7f0c3f6f6640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c380b0390 con 0x7f0c380b4670 2024-03-25T19:19:03.584 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.581+0000 7f0c3f6f6640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c380ae320 con 0x7f0c3813c480 2024-03-25T19:19:03.584 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.581+0000 7f0c3f6f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:03.584 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.581+0000 7f0c3f6f6640 10 monclient(hunting): authenticate will time out at 6247.821690s 2024-03-25T19:19:03.587 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.587+0000 7f0c3cc91640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0c380b4ff0 0x7f0c380ae800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50510/0 (socket says 172.21.15.6:50510) 2024-03-25T19:19:03.587 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.587+0000 7f0c3cc91640 1 -- 172.21.15.6:0/3738756258 learned_addr learned my addr 172.21.15.6:0/3738756258 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:19:03.592 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.592+0000 7f0c2dffb640 10 client.?.objecter ms_handle_connect 0x7f0c380b4ff0 2024-03-25T19:19:03.592 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.592+0000 7f0c2dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:03.593 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.593+0000 7f0c2dffb640 10 client.?.objecter ms_handle_connect 0x7f0c380b4670 2024-03-25T19:19:03.593 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.593+0000 7f0c2dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:03.597 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.597+0000 7f0c2dffb640 1 -- 172.21.15.6:0/3738756258 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 480279197 0 0) 0x7f0c380ab3c0 con 0x7f0c380b4ff0 2024-03-25T19:19:03.597 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.597+0000 7f0c2dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:19:03.597 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.597+0000 7f0c2dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:03.597 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.597+0000 7f0c2dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:03.598 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.597+0000 7f0c2dffb640 10 monclient(hunting): my global_id is 4446 2024-03-25T19:19:03.598 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.597+0000 7f0c2dffb640 1 -- 172.21.15.6:0/3738756258 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c1c004eb0 con 0x7f0c380b4ff0 2024-03-25T19:19:03.610 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.610+0000 7f0c2dffb640 1 -- 172.21.15.6:0/3738756258 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 47114328 0 0) 0x7f0c1c004eb0 con 0x7f0c380b4ff0 2024-03-25T19:19:03.610 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.610+0000 7f0c2dffb640 1 -- 172.21.15.6:0/3738756258 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c380ab3c0 con 0x7f0c380b4ff0 2024-03-25T19:19:03.610 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.610+0000 7f0c2dffb640 1 -- 172.21.15.6:0/3738756258 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f0c28001830 con 0x7f0c380b4ff0 2024-03-25T19:19:03.610 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.610+0000 7f0c2dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:03.611 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.610+0000 7f0c2dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:19:03.611 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.610+0000 7f0c2dffb640 10 monclient(hunting): dump: 2024-03-25T19:19:03.611 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:19:03.611 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:03.611 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:03.611 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:03.611 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:03.611 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:03.611 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:19:03.611 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:19:03.611 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:19:03.611 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:03.611 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.610+0000 7f0c2dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:19:03.617 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.617+0000 7f0c2dffb640 1 -- 172.21.15.6:0/3738756258 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 534073364 0 0) 0x7f0c380b0390 con 0x7f0c380b4670 2024-03-25T19:19:03.617 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.617+0000 7f0c2dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:19:03.617 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.617+0000 7f0c2dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:03.617 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.617+0000 7f0c2dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:03.617 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.617+0000 7f0c2dffb640 10 monclient(hunting): my global_id is 4450 2024-03-25T19:19:03.617 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.617+0000 7f0c2dffb640 1 -- 172.21.15.6:0/3738756258 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c1c004eb0 con 0x7f0c380b4670 2024-03-25T19:19:03.622 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.621+0000 7f0c2dffb640 1 -- 172.21.15.6:0/3738756258 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 756571965 0 0) 0x7f0c380ab3c0 con 0x7f0c380b4ff0 2024-03-25T19:19:03.622 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.621+0000 7f0c2dffb640 1 -- 172.21.15.6:0/3738756258 >> v1:172.21.15.6:6790/0 conn(0x7f0c3813c480 legacy=0x7f0c380aef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:03.622 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.621+0000 7f0c2dffb640 1 -- 172.21.15.6:0/3738756258 >> v1:172.21.15.192:6789/0 conn(0x7f0c380b4670 legacy=0x7f0c380b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:03.622 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.621+0000 7f0c2dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:19:03.622 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.621+0000 7f0c2dffb640 1 monclient: found mon.a 2024-03-25T19:19:03.622 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.621+0000 7f0c2dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:19:03.622 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.621+0000 7f0c2dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:19:03.622 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.621+0000 7f0c2dffb640 1 -- 172.21.15.6:0/3738756258 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c380b0600 con 0x7f0c380b4ff0 2024-03-25T19:19:03.622 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.621+0000 7f0c2dffb640 10 monclient: _finish_auth 0 2024-03-25T19:19:03.622 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.621+0000 7f0c2dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:19:03.622 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.621+0000 7f0c2dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:19:03.622 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.621+0000 7f0c3f6f6640 5 monclient: authenticate success, global_id 4446 2024-03-25T19:19:03.622 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.621+0000 7f0c3f6f6640 10 monclient: _renew_subs 2024-03-25T19:19:03.623 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.621+0000 7f0c3f6f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:19:03.623 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.621+0000 7f0c3f6f6640 1 -- 172.21.15.6:0/3738756258 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c380b08e0 con 0x7f0c380b4ff0 2024-03-25T19:19:03.623 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.621+0000 7f0c3f6f6640 10 client.4446.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:19:03.623 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.621+0000 7f0c3f6f6640 10 monclient: _renew_subs 2024-03-25T19:19:03.623 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.621+0000 7f0c3f6f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:19:03.623 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.621+0000 7f0c3f6f6640 1 -- 172.21.15.6:0/3738756258 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c381e35a0 con 0x7f0c380b4ff0 2024-03-25T19:19:03.623 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.621+0000 7f0c3f6f6640 1 librados: init done 2024-03-25T19:19:03.624 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.622+0000 7f0c3f6f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:03.624 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.622+0000 7f0c3f6f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:03.624 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.623+0000 7f0c3f6f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:19:03.624 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.623+0000 7f0c3f6f6640 1 -- 172.21.15.6:0/3738756258 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c3806dbd0 con 0x7f0c380b4ff0 2024-03-25T19:19:03.839 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.839+0000 7f0c2dffb640 1 -- 172.21.15.6:0/3738756258 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c28003610 con 0x7f0c380b4ff0 2024-03-25T19:19:03.839 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.839+0000 7f0c2dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:19:03.839 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.839+0000 7f0c2f7fe640 4 set_mon_vals no callback set 2024-03-25T19:19:03.948 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.948+0000 7f0c2dffb640 1 -- 172.21.15.6:0/3738756258 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f0c28003a20 con 0x7f0c380b4ff0 2024-03-25T19:19:03.948 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.948+0000 7f0c2dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:03.948 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.948+0000 7f0c2dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:19:03.948 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.948+0000 7f0c2dffb640 10 monclient: dump: 2024-03-25T19:19:03.948 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:19:03.948 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:03.948 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:03.948 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:03.948 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:03.948 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:03.949 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:19:03.949 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:19:03.949 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:19:03.949 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:03.949 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.949+0000 7f0c2dffb640 1 -- 172.21.15.6:0/3738756258 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f0c28003c00 con 0x7f0c380b4ff0 2024-03-25T19:19:03.949 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.949+0000 7f0c3d492640 10 monclient: get_auth_request con 0x7f0c1c06dd50 auth_method 0 2024-03-25T19:19:03.950 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.950+0000 7f0c2dffb640 10 client.4446.objecter ms_handle_connect 0x7f0c1c06dd50 2024-03-25T19:19:03.957 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.957+0000 7f0c2dffb640 1 -- 172.21.15.6:0/3738756258 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f0c28054700 con 0x7f0c380b4ff0 2024-03-25T19:19:03.957 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.957+0000 7f0c2dffb640 10 client.4446.objecter ms_dispatch 0x7f0c38000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:19:03.957 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.957+0000 7f0c2dffb640 3 client.4446.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:19:03.957 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.957+0000 7f0c2dffb640 3 client.4446.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:19:03.957 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.957+0000 7f0c2dffb640 20 client.4446.objecter dump_active .. 0 homeless 2024-03-25T19:19:03.957 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.957+0000 7f0c2dffb640 1 -- 172.21.15.6:0/3738756258 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0c28050210 con 0x7f0c380b4ff0 2024-03-25T19:19:03.957 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.957+0000 7f0c2dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:03.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:03.957+0000 7f0c2dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:19:04.177 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:04.177+0000 7f0c3f6f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:19:04.177 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:04.177+0000 7f0c3f6f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:19:04.177 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:04.177+0000 7f0c3f6f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:19:04.177 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:04.177+0000 7f0c3f6f6640 1 -- 172.21.15.6:0/3738756258 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0c380688c0 con 0x7f0c380b4ff0 2024-03-25T19:19:04.202 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:04.202+0000 7f0c2dffb640 1 -- 172.21.15.6:0/3738756258 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f0c28052140 con 0x7f0c380b4ff0 2024-03-25T19:19:04.203 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:04.202+0000 7f0c2dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:19:04.203 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:04.202+0000 7f0c2dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:19:04.203 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:19:04.203 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:19:04.206 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:04.205+0000 7f0c3f6f6640 10 librados: watch_flush enter 2024-03-25T19:19:04.206 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:04.205+0000 7f0c3f6f6640 10 librados: watch_flush exit 2024-03-25T19:19:04.206 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:04.205+0000 7f0c3f6f6640 20 client.4446.objecter shutdown clearing up homeless session... 2024-03-25T19:19:04.206 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:04.205+0000 7f0c3f6f6640 10 client.4446.objecter successfully canceled tick 2024-03-25T19:19:04.206 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:04.205+0000 7f0c3f6f6640 1 -- 172.21.15.6:0/3738756258 >> v1:172.21.15.6:6800/2936652650 conn(0x7f0c1c06dd50 legacy=0x7f0c1c070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:04.206 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:04.205+0000 7f0c3f6f6640 10 monclient: shutdown 2024-03-25T19:19:04.206 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:04.205+0000 7f0c3f6f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:19:04.206 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:04.205+0000 7f0c3f6f6640 1 -- 172.21.15.6:0/3738756258 >> v1:172.21.15.6:6789/0 conn(0x7f0c380b4ff0 legacy=0x7f0c380ae800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:04.207 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:04.206+0000 7f0c3f6f6640 1 -- 172.21.15.6:0/3738756258 shutdown_connections 2024-03-25T19:19:04.207 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:04.206+0000 7f0c3f6f6640 1 -- 172.21.15.6:0/3738756258 >> 172.21.15.6:0/3738756258 conn(0x7f0c380617a0 msgr2=0x7f0c381407a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:19:04.207 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:04.206+0000 7f0c3f6f6640 1 -- 172.21.15.6:0/3738756258 shutdown_connections 2024-03-25T19:19:04.207 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:04.206+0000 7f0c3f6f6640 1 -- 172.21.15.6:0/3738756258 wait complete. 2024-03-25T19:19:04.207 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:04.206+0000 7f0c3f6f6640 1 librados: shutdown 2024-03-25T19:19:04.219 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:19:11.025 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:19:11.135 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.135+0000 7f76229b6640 10 monclient: get_monmap_and_config 2024-03-25T19:19:11.157 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.135+0000 7f76229b6640 10 monclient: build_initial_monmap 2024-03-25T19:19:11.157 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.135+0000 7f76229b6640 10 monclient: monmap: 2024-03-25T19:19:11.157 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:19:11.158 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:11.158 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:19:11.136082+0000 2024-03-25T19:19:11.158 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:19:11.136082+0000 2024-03-25T19:19:11.158 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:11.158 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:11.158 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:19:11.158 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:19:11.158 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:19:11.158 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:11.160 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.135+0000 7f76229b6640 1 Processor -- start 2024-03-25T19:19:11.160 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.136+0000 7f76229b6640 1 -- start start 2024-03-25T19:19:11.160 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.136+0000 7f76229b6640 10 monclient: init 2024-03-25T19:19:11.160 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.136+0000 7f76229b6640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:11.160 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.136+0000 7f76229b6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:19:11.160 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.136+0000 7f76229b6640 10 monclient(hunting): picked mon.noname-a con 0x7f761c0b4ff0 addr v1:172.21.15.6:6789/0 2024-03-25T19:19:11.160 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.136+0000 7f76229b6640 10 monclient(hunting): picked mon.noname-c con 0x7f761c0b4670 addr v1:172.21.15.192:6789/0 2024-03-25T19:19:11.160 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.136+0000 7f76229b6640 10 monclient(hunting): picked mon.noname-b con 0x7f761c13c480 addr v1:172.21.15.6:6790/0 2024-03-25T19:19:11.160 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.136+0000 7f76229b6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f761c0ab0e0 con 0x7f761c0b4ff0 2024-03-25T19:19:11.160 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.136+0000 7f76229b6640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f761c0ab3c0 con 0x7f761c0b4670 2024-03-25T19:19:11.160 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.136+0000 7f76229b6640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f761c0a9f30 con 0x7f761c13c480 2024-03-25T19:19:11.161 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.136+0000 7f76229b6640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:11.161 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.136+0000 7f76229b6640 10 monclient(hunting): authenticate will time out at 6255.376502s 2024-03-25T19:19:11.161 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.146+0000 7f761b7fe640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f761c0b4670 0x7f761c1425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:52304/0 (socket says 172.21.15.6:52304) 2024-03-25T19:19:11.161 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.146+0000 7f761b7fe640 1 -- 172.21.15.6:0/2025892732 learned_addr learned my addr 172.21.15.6:0/2025892732 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:19:11.161 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.155+0000 7f761affd640 1 -- 172.21.15.6:0/2025892732 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 959675155 0 0) 0x7f761c0ab3c0 con 0x7f761c0b4670 2024-03-25T19:19:11.161 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.155+0000 7f761affd640 10 monclient(hunting): _negotiate 2024-03-25T19:19:11.161 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.155+0000 7f761affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:11.161 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.155+0000 7f761affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:11.161 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.155+0000 7f761affd640 10 monclient(hunting): my global_id is 4453 2024-03-25T19:19:11.161 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.155+0000 7f761affd640 1 -- 172.21.15.6:0/2025892732 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7604003520 con 0x7f761c0b4670 2024-03-25T19:19:11.170 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.169+0000 7f761affd640 1 -- 172.21.15.6:0/2025892732 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3481889872 0 0) 0x7f7604003520 con 0x7f761c0b4670 2024-03-25T19:19:11.170 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.169+0000 7f761affd640 1 -- 172.21.15.6:0/2025892732 >> v1:172.21.15.6:6790/0 conn(0x7f761c13c480 legacy=0x7f761c14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:11.170 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.169+0000 7f761affd640 1 -- 172.21.15.6:0/2025892732 >> v1:172.21.15.6:6789/0 conn(0x7f761c0b4ff0 legacy=0x7f761c13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:11.170 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.169+0000 7f761affd640 10 monclient: _finish_hunting 0 2024-03-25T19:19:11.170 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.169+0000 7f761affd640 1 monclient: found mon.noname-c 2024-03-25T19:19:11.170 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.169+0000 7f761affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:19:11.170 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.169+0000 7f761affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:19:11.170 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.169+0000 7f761affd640 1 -- 172.21.15.6:0/2025892732 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f761c0a9b70 con 0x7f761c0b4670 2024-03-25T19:19:11.170 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.169+0000 7f761affd640 10 monclient: _finish_auth 0 2024-03-25T19:19:11.170 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.169+0000 7f761affd640 10 monclient: _check_auth_tickets 2024-03-25T19:19:11.171 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.169+0000 7f761affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:19:11.171 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.169+0000 7f761affd640 1 -- 172.21.15.6:0/2025892732 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f7610001690 con 0x7f761c0b4670 2024-03-25T19:19:11.171 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.169+0000 7f761affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:11.171 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.169+0000 7f761affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:19:11.171 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.170+0000 7f761affd640 10 monclient: dump: 2024-03-25T19:19:11.171 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:19:11.171 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:11.171 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:11.171 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:11.171 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:11.171 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:11.171 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:19:11.171 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:19:11.171 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:19:11.172 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:11.172 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.170+0000 7f76229b6640 5 monclient: authenticate success, global_id 4453 2024-03-25T19:19:11.172 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.170+0000 7f76229b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:19:11.186 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.186+0000 7f761affd640 1 -- 172.21.15.6:0/2025892732 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76100017f0 con 0x7f761c0b4670 2024-03-25T19:19:11.186 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.186+0000 7f761affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:19:11.186 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.186+0000 7f761affd640 1 -- 172.21.15.6:0/2025892732 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f7610001c40 con 0x7f761c0b4670 2024-03-25T19:19:11.186 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.186+0000 7f76229b6640 10 monclient: get_monmap_and_config success 2024-03-25T19:19:11.187 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.186+0000 7f76229b6640 4 set_mon_vals no callback set 2024-03-25T19:19:11.187 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.186+0000 7f761affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:11.187 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.186+0000 7f761affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:19:11.187 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.186+0000 7f761affd640 10 monclient: dump: 2024-03-25T19:19:11.187 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:19:11.187 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:11.187 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:11.187 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:11.187 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:11.187 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:11.187 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:19:11.187 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:19:11.187 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:19:11.188 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:11.189 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.187+0000 7f76229b6640 10 monclient: shutdown 2024-03-25T19:19:11.189 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.187+0000 7f76229b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:19:11.189 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.187+0000 7f76229b6640 1 -- 172.21.15.6:0/2025892732 >> v1:172.21.15.192:6789/0 conn(0x7f761c0b4670 legacy=0x7f761c1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:11.189 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.187+0000 7f76229b6640 1 -- 172.21.15.6:0/2025892732 shutdown_connections 2024-03-25T19:19:11.189 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.187+0000 7f76229b6640 1 -- 172.21.15.6:0/2025892732 >> 172.21.15.6:0/2025892732 conn(0x7f761c0617a0 msgr2=0x7f761c061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:19:11.189 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.187+0000 7f76229b6640 1 -- 172.21.15.6:0/2025892732 shutdown_connections 2024-03-25T19:19:11.189 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.187+0000 7f76229b6640 1 -- 172.21.15.6:0/2025892732 wait complete. 2024-03-25T19:19:11.189 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.188+0000 7f76229b6640 10 monclient: build_initial_monmap 2024-03-25T19:19:11.190 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.188+0000 7f76229b6640 10 monclient: monmap: 2024-03-25T19:19:11.190 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:19:11.190 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:19:11.190 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:19:11.190 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:19:11.190 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:11.190 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:19:11.190 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:19:11.190 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:19:11.190 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:19:11.190 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:11.190 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.188+0000 7f76229b6640 1 librados: starting msgr at 2024-03-25T19:19:11.190 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.188+0000 7f76229b6640 1 librados: starting objecter 2024-03-25T19:19:11.190 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.189+0000 7f76229b6640 1 Processor -- start 2024-03-25T19:19:11.191 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.189+0000 7f76229b6640 1 -- start start 2024-03-25T19:19:11.191 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.189+0000 7f76229b6640 1 librados: setting wanted keys 2024-03-25T19:19:11.191 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.189+0000 7f76229b6640 1 librados: calling monclient init 2024-03-25T19:19:11.191 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.189+0000 7f76229b6640 10 monclient: init 2024-03-25T19:19:11.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.190+0000 7f76229b6640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:11.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.190+0000 7f76229b6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:19:11.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.190+0000 7f76229b6640 10 monclient(hunting): picked mon.noname-a con 0x7f761c0b4670 addr v1:172.21.15.6:6789/0 2024-03-25T19:19:11.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.190+0000 7f76229b6640 10 monclient(hunting): picked mon.noname-b con 0x7f761c0b4ff0 addr v1:172.21.15.192:6789/0 2024-03-25T19:19:11.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.190+0000 7f76229b6640 10 monclient(hunting): picked mon.noname-c con 0x7f761c13c480 addr v1:172.21.15.6:6790/0 2024-03-25T19:19:11.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.190+0000 7f76229b6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f761c144a50 con 0x7f761c0b4670 2024-03-25T19:19:11.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.190+0000 7f76229b6640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f761c13ec30 con 0x7f761c0b4ff0 2024-03-25T19:19:11.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.190+0000 7f76229b6640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f761c13edc0 con 0x7f761c13c480 2024-03-25T19:19:11.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.190+0000 7f76229b6640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:11.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.190+0000 7f76229b6640 10 monclient(hunting): authenticate will time out at 6255.430065s 2024-03-25T19:19:11.203 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.203+0000 7f7620f53640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f761c13c480 0x7f761c1e26e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:51910/0 (socket says 172.21.15.6:51910) 2024-03-25T19:19:11.204 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.203+0000 7f7620f53640 1 -- 172.21.15.6:0/2493993597 learned_addr learned my addr 172.21.15.6:0/2493993597 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:19:11.204 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.204+0000 7f7618ff9640 10 client.?.objecter ms_handle_connect 0x7f761c13c480 2024-03-25T19:19:11.204 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.204+0000 7f7618ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:11.210 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.210+0000 7f7618ff9640 10 client.?.objecter ms_handle_connect 0x7f761c0b4ff0 2024-03-25T19:19:11.210 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.210+0000 7f7618ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:11.215 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.215+0000 7f7618ff9640 10 client.?.objecter ms_handle_connect 0x7f761c0b4670 2024-03-25T19:19:11.215 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.215+0000 7f7618ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:11.219 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.219+0000 7f7618ff9640 1 -- 172.21.15.6:0/2493993597 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1114347618 0 0) 0x7f761c13edc0 con 0x7f761c13c480 2024-03-25T19:19:11.219 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.219+0000 7f7618ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:19:11.219 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.219+0000 7f7618ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:11.219 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.219+0000 7f7618ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:11.219 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.219+0000 7f7618ff9640 10 monclient(hunting): my global_id is 4451 2024-03-25T19:19:11.219 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.219+0000 7f7618ff9640 1 -- 172.21.15.6:0/2493993597 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7608004eb0 con 0x7f761c13c480 2024-03-25T19:19:11.221 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.221+0000 7f7618ff9640 1 -- 172.21.15.6:0/2493993597 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3212743286 0 0) 0x7f761c13ec30 con 0x7f761c0b4ff0 2024-03-25T19:19:11.221 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.221+0000 7f7618ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:19:11.221 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.221+0000 7f7618ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:11.221 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.221+0000 7f7618ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:11.221 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.221+0000 7f7618ff9640 10 monclient(hunting): my global_id is 4456 2024-03-25T19:19:11.222 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.221+0000 7f7618ff9640 1 -- 172.21.15.6:0/2493993597 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f761c13edc0 con 0x7f761c0b4ff0 2024-03-25T19:19:11.228 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.227+0000 7f7618ff9640 1 -- 172.21.15.6:0/2493993597 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4093439142 0 0) 0x7f7608004eb0 con 0x7f761c13c480 2024-03-25T19:19:11.228 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.228+0000 7f7618ff9640 1 -- 172.21.15.6:0/2493993597 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f761c13ec30 con 0x7f761c13c480 2024-03-25T19:19:11.228 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.228+0000 7f7618ff9640 1 -- 172.21.15.6:0/2493993597 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f7614001c90 con 0x7f761c13c480 2024-03-25T19:19:11.228 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.228+0000 7f7618ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:11.228 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.228+0000 7f7618ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:19:11.228 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.228+0000 7f7618ff9640 10 monclient(hunting): dump: 2024-03-25T19:19:11.228 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:19:11.228 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:11.228 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:11.228 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:11.228 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:11.228 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:11.229 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:19:11.229 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:19:11.229 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:19:11.229 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:11.229 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.228+0000 7f7618ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:19:11.238 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.238+0000 7f7618ff9640 1 -- 172.21.15.6:0/2493993597 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2389263064 0 0) 0x7f761c13ec30 con 0x7f761c13c480 2024-03-25T19:19:11.238 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.238+0000 7f7618ff9640 1 -- 172.21.15.6:0/2493993597 >> v1:172.21.15.192:6789/0 conn(0x7f761c0b4ff0 legacy=0x7f761c145eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:11.239 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.238+0000 7f7618ff9640 1 -- 172.21.15.6:0/2493993597 >> v1:172.21.15.6:6789/0 conn(0x7f761c0b4670 legacy=0x7f761c1457a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:11.239 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.238+0000 7f7618ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:19:11.239 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.238+0000 7f7618ff9640 1 monclient: found mon.c 2024-03-25T19:19:11.239 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.238+0000 7f7618ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:19:11.239 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.238+0000 7f7618ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:19:11.239 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.238+0000 7f7618ff9640 1 -- 172.21.15.6:0/2493993597 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f761c13f6e0 con 0x7f761c13c480 2024-03-25T19:19:11.239 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.238+0000 7f7618ff9640 10 monclient: _finish_auth 0 2024-03-25T19:19:11.239 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.238+0000 7f7618ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:19:11.239 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.238+0000 7f7618ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:19:11.239 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.238+0000 7f76229b6640 5 monclient: authenticate success, global_id 4451 2024-03-25T19:19:11.239 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.238+0000 7f76229b6640 10 monclient: _renew_subs 2024-03-25T19:19:11.239 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.238+0000 7f76229b6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:19:11.239 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.238+0000 7f76229b6640 1 -- 172.21.15.6:0/2493993597 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f761c13f930 con 0x7f761c13c480 2024-03-25T19:19:11.239 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.238+0000 7f76229b6640 10 client.4451.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:19:11.239 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.238+0000 7f76229b6640 10 monclient: _renew_subs 2024-03-25T19:19:11.240 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.238+0000 7f76229b6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:19:11.240 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.238+0000 7f76229b6640 1 -- 172.21.15.6:0/2493993597 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f761c143ab0 con 0x7f761c13c480 2024-03-25T19:19:11.240 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.238+0000 7f76229b6640 1 librados: init done 2024-03-25T19:19:11.240 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.239+0000 7f76229b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:11.240 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.239+0000 7f76229b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:11.240 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.239+0000 7f76229b6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:19:11.240 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.239+0000 7f76229b6640 1 -- 172.21.15.6:0/2493993597 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f761c145150 con 0x7f761c13c480 2024-03-25T19:19:11.558 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.557+0000 7f7618ff9640 1 -- 172.21.15.6:0/2493993597 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7614003990 con 0x7f761c13c480 2024-03-25T19:19:11.558 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.557+0000 7f7618ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:19:11.558 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.557+0000 7f7618ff9640 1 -- 172.21.15.6:0/2493993597 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f7614003b40 con 0x7f761c13c480 2024-03-25T19:19:11.558 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.557+0000 7f7618ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:11.558 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.557+0000 7f7618ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:19:11.558 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.557+0000 7f7618ff9640 10 monclient: dump: 2024-03-25T19:19:11.558 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:19:11.558 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:11.559 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:11.559 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:11.559 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:11.559 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:11.559 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:19:11.559 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:19:11.559 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:19:11.559 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:11.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.557+0000 7f761a7fc640 4 set_mon_vals no callback set 2024-03-25T19:19:11.560 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.558+0000 7f7618ff9640 1 -- 172.21.15.6:0/2493993597 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f7614003d60 con 0x7f761c13c480 2024-03-25T19:19:11.560 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.559+0000 7f761bfff640 10 monclient: get_auth_request con 0x7f760806dd10 auth_method 0 2024-03-25T19:19:11.560 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.559+0000 7f7618ff9640 10 client.4451.objecter ms_handle_connect 0x7f760806dd10 2024-03-25T19:19:11.566 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.566+0000 7f7618ff9640 1 -- 172.21.15.6:0/2493993597 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f7614054890 con 0x7f761c13c480 2024-03-25T19:19:11.566 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.566+0000 7f7618ff9640 10 client.4451.objecter ms_dispatch 0x7f761c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:19:11.566 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.566+0000 7f7618ff9640 3 client.4451.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:19:11.566 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.566+0000 7f7618ff9640 3 client.4451.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:19:11.566 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.566+0000 7f7618ff9640 20 client.4451.objecter dump_active .. 0 homeless 2024-03-25T19:19:11.566 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.566+0000 7f7618ff9640 1 -- 172.21.15.6:0/2493993597 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f76140503a0 con 0x7f761c13c480 2024-03-25T19:19:11.566 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.566+0000 7f7618ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:11.566 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.566+0000 7f7618ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:19:11.804 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.804+0000 7f76229b6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:19:11.804 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.804+0000 7f76229b6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:19:11.804 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.804+0000 7f76229b6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:19:11.804 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.804+0000 7f76229b6640 1 -- 172.21.15.6:0/2493993597 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f761c06dbd0 con 0x7f761c13c480 2024-03-25T19:19:11.826 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.826+0000 7f7618ff9640 1 -- 172.21.15.6:0/2493993597 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f76140522d0 con 0x7f761c13c480 2024-03-25T19:19:11.826 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.826+0000 7f7618ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:19:11.826 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.826+0000 7f7618ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:19:11.827 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:19:11.827 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:19:11.829 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.828+0000 7f76229b6640 10 librados: watch_flush enter 2024-03-25T19:19:11.829 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.828+0000 7f76229b6640 10 librados: watch_flush exit 2024-03-25T19:19:11.829 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.828+0000 7f76229b6640 20 client.4451.objecter shutdown clearing up homeless session... 2024-03-25T19:19:11.829 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.828+0000 7f76229b6640 10 client.4451.objecter successfully canceled tick 2024-03-25T19:19:11.830 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.829+0000 7f76229b6640 1 -- 172.21.15.6:0/2493993597 >> v1:172.21.15.6:6800/2936652650 conn(0x7f760806dd10 legacy=0x7f76080701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:11.830 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.829+0000 7f76229b6640 10 monclient: shutdown 2024-03-25T19:19:11.830 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.829+0000 7f76229b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:19:11.830 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.829+0000 7f76229b6640 1 -- 172.21.15.6:0/2493993597 >> v1:172.21.15.6:6790/0 conn(0x7f761c13c480 legacy=0x7f761c1e26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:11.830 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.829+0000 7f76229b6640 1 -- 172.21.15.6:0/2493993597 shutdown_connections 2024-03-25T19:19:11.830 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.829+0000 7f76229b6640 1 -- 172.21.15.6:0/2493993597 >> 172.21.15.6:0/2493993597 conn(0x7f761c0617a0 msgr2=0x7f761c1407f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:19:11.830 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.829+0000 7f76229b6640 1 -- 172.21.15.6:0/2493993597 shutdown_connections 2024-03-25T19:19:11.830 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.829+0000 7f76229b6640 1 -- 172.21.15.6:0/2493993597 wait complete. 2024-03-25T19:19:11.830 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:11.829+0000 7f76229b6640 1 librados: shutdown 2024-03-25T19:19:11.841 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:19:18.649 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:19:18.768 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.767+0000 7f822caa8640 10 monclient: get_monmap_and_config 2024-03-25T19:19:18.768 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.767+0000 7f822caa8640 10 monclient: build_initial_monmap 2024-03-25T19:19:18.768 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.767+0000 7f822caa8640 10 monclient: monmap: 2024-03-25T19:19:18.768 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:19:18.768 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:18.768 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:19:18.768851+0000 2024-03-25T19:19:18.768 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:19:18.768851+0000 2024-03-25T19:19:18.768 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:18.768 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:18.768 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:19:18.768 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:19:18.769 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:19:18.769 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:18.770 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.768+0000 7f822caa8640 1 Processor -- start 2024-03-25T19:19:18.770 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.768+0000 7f822caa8640 1 -- start start 2024-03-25T19:19:18.770 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.768+0000 7f822caa8640 10 monclient: init 2024-03-25T19:19:18.770 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.769+0000 7f822caa8640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:18.770 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.769+0000 7f822caa8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:19:18.770 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.769+0000 7f822caa8640 10 monclient(hunting): picked mon.noname-c con 0x7f82280b4ff0 addr v1:172.21.15.192:6789/0 2024-03-25T19:19:18.770 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.769+0000 7f822caa8640 10 monclient(hunting): picked mon.noname-b con 0x7f82280b4670 addr v1:172.21.15.6:6790/0 2024-03-25T19:19:18.770 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.769+0000 7f822caa8640 10 monclient(hunting): picked mon.noname-a con 0x7f822813c480 addr v1:172.21.15.6:6789/0 2024-03-25T19:19:18.771 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.769+0000 7f822caa8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82280ab0e0 con 0x7f822813c480 2024-03-25T19:19:18.771 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.769+0000 7f822caa8640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82280ab3c0 con 0x7f82280b4ff0 2024-03-25T19:19:18.771 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.769+0000 7f822caa8640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82280a9f30 con 0x7f82280b4670 2024-03-25T19:19:18.771 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.769+0000 7f822caa8640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:18.771 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.769+0000 7f822caa8640 10 monclient(hunting): authenticate will time out at 6263.009314s 2024-03-25T19:19:18.775 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.775+0000 7f822659c640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f82280b4ff0 0x7f822813bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:60924/0 (socket says 172.21.15.6:60924) 2024-03-25T19:19:18.775 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.775+0000 7f822659c640 1 -- 172.21.15.6:0/597518092 learned_addr learned my addr 172.21.15.6:0/597518092 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:19:18.793 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.793+0000 7f822559a640 1 -- 172.21.15.6:0/597518092 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 468369067 0 0) 0x7f82280ab3c0 con 0x7f82280b4ff0 2024-03-25T19:19:18.793 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.793+0000 7f822559a640 10 monclient(hunting): _negotiate 2024-03-25T19:19:18.793 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.793+0000 7f822559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:18.794 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.793+0000 7f822559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:18.794 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.793+0000 7f822559a640 10 monclient(hunting): my global_id is 4459 2024-03-25T19:19:18.794 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.793+0000 7f822559a640 1 -- 172.21.15.6:0/597518092 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8210003520 con 0x7f82280b4ff0 2024-03-25T19:19:18.811 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.811+0000 7f822559a640 1 -- 172.21.15.6:0/597518092 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 595219742 0 0) 0x7f82280ab0e0 con 0x7f822813c480 2024-03-25T19:19:18.812 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.811+0000 7f822559a640 10 monclient(hunting): _negotiate 2024-03-25T19:19:18.812 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.811+0000 7f822559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:18.812 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.811+0000 7f822559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:18.812 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.811+0000 7f822559a640 10 monclient(hunting): my global_id is 4455 2024-03-25T19:19:18.812 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.811+0000 7f822559a640 1 -- 172.21.15.6:0/597518092 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82280ab3c0 con 0x7f822813c480 2024-03-25T19:19:18.813 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.813+0000 7f822559a640 1 -- 172.21.15.6:0/597518092 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2356019284 0 0) 0x7f82280a9f30 con 0x7f82280b4670 2024-03-25T19:19:18.813 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.813+0000 7f822559a640 10 monclient(hunting): _negotiate 2024-03-25T19:19:18.813 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.813+0000 7f822559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:18.813 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.813+0000 7f822559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:18.813 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.813+0000 7f822559a640 10 monclient(hunting): my global_id is 4454 2024-03-25T19:19:18.813 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.813+0000 7f822559a640 1 -- 172.21.15.6:0/597518092 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82280ab0e0 con 0x7f82280b4670 2024-03-25T19:19:18.820 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.819+0000 7f822559a640 1 -- 172.21.15.6:0/597518092 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2674879125 0 0) 0x7f8210003520 con 0x7f82280b4ff0 2024-03-25T19:19:18.820 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.819+0000 7f822559a640 1 -- 172.21.15.6:0/597518092 >> v1:172.21.15.6:6790/0 conn(0x7f82280b4670 legacy=0x7f82281425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:18.820 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.819+0000 7f822559a640 1 -- 172.21.15.6:0/597518092 >> v1:172.21.15.6:6789/0 conn(0x7f822813c480 legacy=0x7f822814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:18.820 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.819+0000 7f822559a640 10 monclient: _finish_hunting 0 2024-03-25T19:19:18.820 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.819+0000 7f822559a640 1 monclient: found mon.noname-c 2024-03-25T19:19:18.820 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.819+0000 7f822559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:19:18.820 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.819+0000 7f822559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:19:18.820 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.819+0000 7f822559a640 1 -- 172.21.15.6:0/597518092 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82280a9b70 con 0x7f82280b4ff0 2024-03-25T19:19:18.821 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.819+0000 7f822559a640 10 monclient: _finish_auth 0 2024-03-25T19:19:18.821 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.819+0000 7f822559a640 10 monclient: _check_auth_tickets 2024-03-25T19:19:18.821 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.819+0000 7f822559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:19:18.821 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.819+0000 7f822559a640 1 -- 172.21.15.6:0/597518092 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f8218001690 con 0x7f82280b4ff0 2024-03-25T19:19:18.821 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.819+0000 7f822559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:18.821 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.819+0000 7f822559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:19:18.821 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.819+0000 7f822559a640 10 monclient: dump: 2024-03-25T19:19:18.821 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:19:18.821 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:18.821 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:18.821 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:18.821 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:18.821 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:18.821 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:19:18.821 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:19:18.822 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:19:18.822 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:18.822 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.820+0000 7f822caa8640 5 monclient: authenticate success, global_id 4459 2024-03-25T19:19:18.822 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.820+0000 7f822caa8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:19:18.834 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.834+0000 7f822559a640 1 -- 172.21.15.6:0/597518092 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82180017f0 con 0x7f82280b4ff0 2024-03-25T19:19:18.834 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.834+0000 7f822559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:19:18.834 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.834+0000 7f822559a640 1 -- 172.21.15.6:0/597518092 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f8218001c00 con 0x7f82280b4ff0 2024-03-25T19:19:18.835 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.834+0000 7f822559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:18.835 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.834+0000 7f822559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:19:18.835 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.834+0000 7f822559a640 10 monclient: dump: 2024-03-25T19:19:18.835 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:19:18.835 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:18.835 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:18.835 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:18.835 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:18.835 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:18.835 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:19:18.835 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:19:18.835 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:19:18.835 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:18.836 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.834+0000 7f822caa8640 10 monclient: get_monmap_and_config success 2024-03-25T19:19:18.836 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.834+0000 7f822caa8640 4 set_mon_vals no callback set 2024-03-25T19:19:18.836 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.835+0000 7f822caa8640 10 monclient: shutdown 2024-03-25T19:19:18.837 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.835+0000 7f822caa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:19:18.837 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.835+0000 7f822caa8640 1 -- 172.21.15.6:0/597518092 >> v1:172.21.15.192:6789/0 conn(0x7f82280b4ff0 legacy=0x7f822813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:18.837 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.835+0000 7f822caa8640 1 -- 172.21.15.6:0/597518092 shutdown_connections 2024-03-25T19:19:18.837 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.835+0000 7f822caa8640 1 -- 172.21.15.6:0/597518092 >> 172.21.15.6:0/597518092 conn(0x7f82280617a0 msgr2=0x7f8228061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:19:18.837 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.835+0000 7f822caa8640 1 -- 172.21.15.6:0/597518092 shutdown_connections 2024-03-25T19:19:18.837 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.835+0000 7f822caa8640 1 -- 172.21.15.6:0/597518092 wait complete. 2024-03-25T19:19:18.837 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.837+0000 7f822caa8640 10 monclient: build_initial_monmap 2024-03-25T19:19:18.837 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.837+0000 7f822caa8640 10 monclient: monmap: 2024-03-25T19:19:18.837 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:19:18.837 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:19:18.837 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:19:18.837 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:19:18.837 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:18.838 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:19:18.838 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:19:18.838 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:19:18.838 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:19:18.838 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:18.842 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.837+0000 7f822caa8640 1 librados: starting msgr at 2024-03-25T19:19:18.842 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.837+0000 7f822caa8640 1 librados: starting objecter 2024-03-25T19:19:18.842 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.837+0000 7f822caa8640 1 Processor -- start 2024-03-25T19:19:18.842 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.837+0000 7f822caa8640 1 -- start start 2024-03-25T19:19:18.842 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.837+0000 7f822caa8640 1 librados: setting wanted keys 2024-03-25T19:19:18.842 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.837+0000 7f822caa8640 1 librados: calling monclient init 2024-03-25T19:19:18.842 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.837+0000 7f822caa8640 10 monclient: init 2024-03-25T19:19:18.842 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.838+0000 7f822caa8640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:18.843 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.838+0000 7f822caa8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:19:18.843 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.838+0000 7f822caa8640 10 monclient(hunting): picked mon.noname-a con 0x7f82280b4670 addr v1:172.21.15.6:6789/0 2024-03-25T19:19:18.843 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.838+0000 7f822caa8640 10 monclient(hunting): picked mon.noname-c con 0x7f82280b4ff0 addr v1:172.21.15.6:6790/0 2024-03-25T19:19:18.843 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.838+0000 7f822caa8640 10 monclient(hunting): picked mon.noname-b con 0x7f822813c480 addr v1:172.21.15.192:6789/0 2024-03-25T19:19:18.843 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.838+0000 7f822caa8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8228144490 con 0x7f82280b4670 2024-03-25T19:19:18.843 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.838+0000 7f822caa8640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8228130ad0 con 0x7f822813c480 2024-03-25T19:19:18.843 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.838+0000 7f822caa8640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8228144700 con 0x7f82280b4ff0 2024-03-25T19:19:18.843 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.838+0000 7f822caa8640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:18.843 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.838+0000 7f822caa8640 10 monclient(hunting): authenticate will time out at 6263.078527s 2024-03-25T19:19:18.851 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.851+0000 7f8225d9b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f82280b4ff0 0x7f822812fcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:39568/0 (socket says 172.21.15.6:39568) 2024-03-25T19:19:18.851 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.851+0000 7f8225d9b640 1 -- 172.21.15.6:0/3961134519 learned_addr learned my addr 172.21.15.6:0/3961134519 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:19:18.852 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.852+0000 7f8216ffd640 10 client.?.objecter ms_handle_connect 0x7f82280b4ff0 2024-03-25T19:19:18.852 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.852+0000 7f8216ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:18.854 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.854+0000 7f8216ffd640 10 client.?.objecter ms_handle_connect 0x7f822813c480 2024-03-25T19:19:18.854 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.854+0000 7f8216ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:18.857 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.857+0000 7f8216ffd640 10 client.?.objecter ms_handle_connect 0x7f82280b4670 2024-03-25T19:19:18.858 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.857+0000 7f8216ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:18.859 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.859+0000 7f8216ffd640 1 -- 172.21.15.6:0/3961134519 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 419118164 0 0) 0x7f8228144700 con 0x7f82280b4ff0 2024-03-25T19:19:18.860 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.859+0000 7f8216ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:19:18.860 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.859+0000 7f8216ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:18.860 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.859+0000 7f8216ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:18.860 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.859+0000 7f8216ffd640 10 monclient(hunting): my global_id is 4457 2024-03-25T19:19:18.860 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.859+0000 7f8216ffd640 1 -- 172.21.15.6:0/3961134519 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8208004eb0 con 0x7f82280b4ff0 2024-03-25T19:19:18.866 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.866+0000 7f8216ffd640 1 -- 172.21.15.6:0/3961134519 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 90551167 0 0) 0x7f8228130ad0 con 0x7f822813c480 2024-03-25T19:19:18.866 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.866+0000 7f8216ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:19:18.866 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.866+0000 7f8216ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:18.866 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.866+0000 7f8216ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:18.866 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.866+0000 7f8216ffd640 10 monclient(hunting): my global_id is 4462 2024-03-25T19:19:18.866 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.866+0000 7f8216ffd640 1 -- 172.21.15.6:0/3961134519 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8228144700 con 0x7f822813c480 2024-03-25T19:19:18.869 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.869+0000 7f8216ffd640 1 -- 172.21.15.6:0/3961134519 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 716518399 0 0) 0x7f8208004eb0 con 0x7f82280b4ff0 2024-03-25T19:19:18.869 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.869+0000 7f8216ffd640 1 -- 172.21.15.6:0/3961134519 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8228130ad0 con 0x7f82280b4ff0 2024-03-25T19:19:18.870 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.869+0000 7f8216ffd640 1 -- 172.21.15.6:0/3961134519 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f821c001a10 con 0x7f82280b4ff0 2024-03-25T19:19:18.870 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.869+0000 7f8216ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:18.870 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.869+0000 7f8216ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:19:18.870 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.869+0000 7f8216ffd640 10 monclient(hunting): dump: 2024-03-25T19:19:18.870 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:19:18.870 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:18.870 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:18.870 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:18.870 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:18.870 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:18.870 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:19:18.870 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:19:18.871 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:19:18.871 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:18.871 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.869+0000 7f8216ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:19:18.871 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.871+0000 7f8216ffd640 1 -- 172.21.15.6:0/3961134519 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 872063051 0 0) 0x7f8228144490 con 0x7f82280b4670 2024-03-25T19:19:18.871 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.871+0000 7f8216ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:19:18.871 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.871+0000 7f8216ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:18.872 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.871+0000 7f8216ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:18.872 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.871+0000 7f8216ffd640 10 monclient(hunting): my global_id is 4458 2024-03-25T19:19:18.872 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.871+0000 7f8216ffd640 1 -- 172.21.15.6:0/3961134519 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8208004eb0 con 0x7f82280b4670 2024-03-25T19:19:18.883 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.883+0000 7f8216ffd640 1 -- 172.21.15.6:0/3961134519 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2689539097 0 0) 0x7f8228130ad0 con 0x7f82280b4ff0 2024-03-25T19:19:18.883 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.883+0000 7f8216ffd640 1 -- 172.21.15.6:0/3961134519 >> v1:172.21.15.192:6789/0 conn(0x7f822813c480 legacy=0x7f82281303c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:18.884 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.883+0000 7f8216ffd640 1 -- 172.21.15.6:0/3961134519 >> v1:172.21.15.6:6789/0 conn(0x7f82280b4670 legacy=0x7f82281450e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:18.884 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.883+0000 7f8216ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:19:18.884 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.883+0000 7f8216ffd640 1 monclient: found mon.c 2024-03-25T19:19:18.884 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.883+0000 7f8216ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:19:18.884 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.883+0000 7f8216ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:19:18.884 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.883+0000 7f8216ffd640 1 -- 172.21.15.6:0/3961134519 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82281449e0 con 0x7f82280b4ff0 2024-03-25T19:19:18.884 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.883+0000 7f8216ffd640 10 monclient: _finish_auth 0 2024-03-25T19:19:18.884 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.883+0000 7f8216ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:19:18.884 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.883+0000 7f8216ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:19:18.884 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.883+0000 7f822caa8640 5 monclient: authenticate success, global_id 4457 2024-03-25T19:19:18.884 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.883+0000 7f822caa8640 10 monclient: _renew_subs 2024-03-25T19:19:18.884 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.883+0000 7f822caa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:19:18.884 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.883+0000 7f822caa8640 1 -- 172.21.15.6:0/3961134519 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8228144cc0 con 0x7f82280b4ff0 2024-03-25T19:19:18.884 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.883+0000 7f822caa8640 10 client.4457.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:19:18.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.883+0000 7f822caa8640 10 monclient: _renew_subs 2024-03-25T19:19:18.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.883+0000 7f822caa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:19:18.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.883+0000 7f822caa8640 1 -- 172.21.15.6:0/3961134519 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8228143cd0 con 0x7f82280b4ff0 2024-03-25T19:19:18.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.883+0000 7f822caa8640 1 librados: init done 2024-03-25T19:19:18.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.884+0000 7f822caa8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:18.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.884+0000 7f822caa8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:18.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.884+0000 7f822caa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:19:18.886 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:18.884+0000 7f822caa8640 1 -- 172.21.15.6:0/3961134519 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8228143810 con 0x7f82280b4ff0 2024-03-25T19:19:19.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:19.192+0000 7f8216ffd640 1 -- 172.21.15.6:0/3961134519 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f821c001c70 con 0x7f82280b4ff0 2024-03-25T19:19:19.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:19.192+0000 7f8216ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:19:19.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:19.192+0000 7f8216ffd640 1 -- 172.21.15.6:0/3961134519 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f821c0038c0 con 0x7f82280b4ff0 2024-03-25T19:19:19.193 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:19.192+0000 7f8216ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:19.193 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:19.192+0000 7f8216ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:19:19.193 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:19.192+0000 7f8216ffd640 10 monclient: dump: 2024-03-25T19:19:19.193 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:19:19.193 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:19.193 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:19.193 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:19.193 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:19.193 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:19.193 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:19:19.193 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:19:19.194 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:19:19.194 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:19.194 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:19.192+0000 7f8224d99640 4 set_mon_vals no callback set 2024-03-25T19:19:19.194 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:19.193+0000 7f8216ffd640 1 -- 172.21.15.6:0/3961134519 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f821c003ae0 con 0x7f82280b4ff0 2024-03-25T19:19:19.194 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:19.193+0000 7f822659c640 10 monclient: get_auth_request con 0x7f820806de00 auth_method 0 2024-03-25T19:19:19.194 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:19.193+0000 7f8216ffd640 10 client.4457.objecter ms_handle_connect 0x7f820806de00 2024-03-25T19:19:19.200 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:19.199+0000 7f8216ffd640 1 -- 172.21.15.6:0/3961134519 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f821c054680 con 0x7f82280b4ff0 2024-03-25T19:19:19.200 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:19.199+0000 7f8216ffd640 10 client.4457.objecter ms_dispatch 0x7f8228000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:19:19.200 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:19.199+0000 7f8216ffd640 3 client.4457.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:19:19.200 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:19.199+0000 7f8216ffd640 3 client.4457.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:19:19.200 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:19.199+0000 7f8216ffd640 20 client.4457.objecter dump_active .. 0 homeless 2024-03-25T19:19:19.200 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:19.199+0000 7f8216ffd640 1 -- 172.21.15.6:0/3961134519 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f821c050190 con 0x7f82280b4ff0 2024-03-25T19:19:19.200 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:19.199+0000 7f8216ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:19.200 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:19.199+0000 7f8216ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:19:19.438 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:19.437+0000 7f822caa8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:19:19.438 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:19.437+0000 7f822caa8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:19:19.438 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:19.437+0000 7f822caa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:19:19.438 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:19.437+0000 7f822caa8640 1 -- 172.21.15.6:0/3961134519 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f822806dbd0 con 0x7f82280b4ff0 2024-03-25T19:19:19.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:19.471+0000 7f8216ffd640 1 -- 172.21.15.6:0/3961134519 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f821c0520c0 con 0x7f82280b4ff0 2024-03-25T19:19:19.472 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:19.471+0000 7f8216ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:19:19.472 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:19.471+0000 7f8216ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:19:19.472 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:19:19.472 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:19:19.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:19.473+0000 7f822caa8640 10 librados: watch_flush enter 2024-03-25T19:19:19.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:19.473+0000 7f822caa8640 10 librados: watch_flush exit 2024-03-25T19:19:19.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:19.473+0000 7f822caa8640 20 client.4457.objecter shutdown clearing up homeless session... 2024-03-25T19:19:19.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:19.474+0000 7f822caa8640 10 client.4457.objecter successfully canceled tick 2024-03-25T19:19:19.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:19.474+0000 7f822caa8640 1 -- 172.21.15.6:0/3961134519 >> v1:172.21.15.6:6800/2936652650 conn(0x7f820806de00 legacy=0x7f82080702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:19.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:19.474+0000 7f822caa8640 10 monclient: shutdown 2024-03-25T19:19:19.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:19.474+0000 7f822caa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:19:19.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:19.474+0000 7f822caa8640 1 -- 172.21.15.6:0/3961134519 >> v1:172.21.15.6:6790/0 conn(0x7f82280b4ff0 legacy=0x7f822812fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:19.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:19.474+0000 7f822caa8640 1 -- 172.21.15.6:0/3961134519 shutdown_connections 2024-03-25T19:19:19.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:19.474+0000 7f822caa8640 1 -- 172.21.15.6:0/3961134519 >> 172.21.15.6:0/3961134519 conn(0x7f82280617a0 msgr2=0x7f82280b6fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:19:19.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:19.474+0000 7f822caa8640 1 -- 172.21.15.6:0/3961134519 shutdown_connections 2024-03-25T19:19:19.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:19.474+0000 7f822caa8640 1 -- 172.21.15.6:0/3961134519 wait complete. 2024-03-25T19:19:19.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:19.474+0000 7f822caa8640 1 librados: shutdown 2024-03-25T19:19:19.488 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:19:23.421 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:05:40.767 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:19:23.431 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:05:40.778 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:19:23.461 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:05:40.808 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:19:23.462 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:05:40.780 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:19:23.475 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:05:40.792 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:19:23.480 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:05:40.827 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:19:23.489 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:05:40.836 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:19:23.495 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:05:40.813 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:19:23.518 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:05:40.865 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:19:26.295 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:19:26.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.416+0000 7f236ae63640 10 monclient: get_monmap_and_config 2024-03-25T19:19:26.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.416+0000 7f236ae63640 10 monclient: build_initial_monmap 2024-03-25T19:19:26.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.416+0000 7f236ae63640 10 monclient: monmap: 2024-03-25T19:19:26.416 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:19:26.416 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:26.416 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:19:26.417158+0000 2024-03-25T19:19:26.416 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:19:26.417158+0000 2024-03-25T19:19:26.417 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:26.417 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:26.417 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:19:26.417 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:19:26.417 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:19:26.417 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:26.418 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.417+0000 7f236ae63640 1 Processor -- start 2024-03-25T19:19:26.418 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.417+0000 7f236ae63640 1 -- start start 2024-03-25T19:19:26.418 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.417+0000 7f236ae63640 10 monclient: init 2024-03-25T19:19:26.418 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.418+0000 7f236ae63640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:26.418 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.418+0000 7f236ae63640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:19:26.419 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.418+0000 7f236ae63640 10 monclient(hunting): picked mon.noname-b con 0x7f23640b4e70 addr v1:172.21.15.6:6790/0 2024-03-25T19:19:26.419 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.418+0000 7f236ae63640 10 monclient(hunting): picked mon.noname-c con 0x7f23640b44b0 addr v1:172.21.15.192:6789/0 2024-03-25T19:19:26.419 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.418+0000 7f236ae63640 10 monclient(hunting): picked mon.noname-a con 0x7f23640b1b00 addr v1:172.21.15.6:6789/0 2024-03-25T19:19:26.419 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.418+0000 7f236ae63640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23640a96f0 con 0x7f23640b1b00 2024-03-25T19:19:26.419 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.418+0000 7f236ae63640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23640ab0e0 con 0x7f23640b44b0 2024-03-25T19:19:26.419 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.418+0000 7f236ae63640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2364145f70 con 0x7f23640b4e70 2024-03-25T19:19:26.419 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.418+0000 7f236ae63640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:26.419 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.418+0000 7f236ae63640 10 monclient(hunting): authenticate will time out at 6270.658024s 2024-03-25T19:19:26.426 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.426+0000 7f2363fff640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f23640b44b0 0x7f2364145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:37348/0 (socket says 172.21.15.6:37348) 2024-03-25T19:19:26.426 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.427+0000 7f2363fff640 1 -- 172.21.15.6:0/507591245 learned_addr learned my addr 172.21.15.6:0/507591245 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:19:26.440 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.440+0000 7f23637fe640 1 -- 172.21.15.6:0/507591245 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1791669899 0 0) 0x7f23640ab0e0 con 0x7f23640b44b0 2024-03-25T19:19:26.440 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.440+0000 7f23637fe640 10 monclient(hunting): _negotiate 2024-03-25T19:19:26.440 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.440+0000 7f23637fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:26.440 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.440+0000 7f23637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:26.440 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.440+0000 7f23637fe640 10 monclient(hunting): my global_id is 4465 2024-03-25T19:19:26.440 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.440+0000 7f23637fe640 1 -- 172.21.15.6:0/507591245 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2354003520 con 0x7f23640b44b0 2024-03-25T19:19:26.453 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.453+0000 7f23637fe640 1 -- 172.21.15.6:0/507591245 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3866522939 0 0) 0x7f23640a96f0 con 0x7f23640b1b00 2024-03-25T19:19:26.453 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.453+0000 7f23637fe640 10 monclient(hunting): _negotiate 2024-03-25T19:19:26.453 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.453+0000 7f23637fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:26.453 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.453+0000 7f23637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:26.453 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.453+0000 7f23637fe640 10 monclient(hunting): my global_id is 4461 2024-03-25T19:19:26.453 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.453+0000 7f23637fe640 1 -- 172.21.15.6:0/507591245 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23640ab0e0 con 0x7f23640b1b00 2024-03-25T19:19:26.455 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.454+0000 7f23637fe640 1 -- 172.21.15.6:0/507591245 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1720643630 0 0) 0x7f2354003520 con 0x7f23640b44b0 2024-03-25T19:19:26.455 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.454+0000 7f23637fe640 1 -- 172.21.15.6:0/507591245 >> v1:172.21.15.6:6790/0 conn(0x7f23640b4e70 legacy=0x7f236413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:26.455 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.454+0000 7f23637fe640 1 -- 172.21.15.6:0/507591245 >> v1:172.21.15.6:6789/0 conn(0x7f23640b1b00 legacy=0x7f236414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:26.455 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.454+0000 7f23637fe640 10 monclient: _finish_hunting 0 2024-03-25T19:19:26.455 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.454+0000 7f23637fe640 1 monclient: found mon.noname-c 2024-03-25T19:19:26.455 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.454+0000 7f23637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:19:26.455 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.454+0000 7f23637fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:19:26.455 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.454+0000 7f23637fe640 1 -- 172.21.15.6:0/507591245 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23640a9b80 con 0x7f23640b44b0 2024-03-25T19:19:26.455 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.454+0000 7f23637fe640 10 monclient: _finish_auth 0 2024-03-25T19:19:26.455 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.454+0000 7f23637fe640 10 monclient: _check_auth_tickets 2024-03-25T19:19:26.455 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.454+0000 7f23637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:19:26.455 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.454+0000 7f23637fe640 1 -- 172.21.15.6:0/507591245 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f2350001690 con 0x7f23640b44b0 2024-03-25T19:19:26.455 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.454+0000 7f23637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:26.456 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.454+0000 7f23637fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:19:26.456 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.454+0000 7f23637fe640 10 monclient: dump: 2024-03-25T19:19:26.456 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:19:26.456 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:26.456 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:26.456 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:26.456 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:26.456 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:26.456 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:19:26.456 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:19:26.456 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:19:26.456 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:26.456 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.454+0000 7f236ae63640 5 monclient: authenticate success, global_id 4465 2024-03-25T19:19:26.456 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.454+0000 7f236ae63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:19:26.468 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.468+0000 7f23637fe640 1 -- 172.21.15.6:0/507591245 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23500017f0 con 0x7f23640b44b0 2024-03-25T19:19:26.468 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.468+0000 7f23637fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:19:26.468 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.468+0000 7f23637fe640 1 -- 172.21.15.6:0/507591245 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f2350001c00 con 0x7f23640b44b0 2024-03-25T19:19:26.468 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.468+0000 7f236ae63640 10 monclient: get_monmap_and_config success 2024-03-25T19:19:26.468 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.468+0000 7f236ae63640 4 set_mon_vals no callback set 2024-03-25T19:19:26.468 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.468+0000 7f23637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:26.468 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.468+0000 7f23637fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:19:26.468 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.468+0000 7f23637fe640 10 monclient: dump: 2024-03-25T19:19:26.468 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:19:26.468 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:26.469 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:26.469 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:26.469 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:26.469 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:26.469 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:19:26.469 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:19:26.469 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:19:26.469 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:26.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.468+0000 7f236ae63640 10 monclient: shutdown 2024-03-25T19:19:26.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.468+0000 7f236ae63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:19:26.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.468+0000 7f236ae63640 1 -- 172.21.15.6:0/507591245 >> v1:172.21.15.192:6789/0 conn(0x7f23640b44b0 legacy=0x7f2364145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:26.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.469+0000 7f236ae63640 1 -- 172.21.15.6:0/507591245 shutdown_connections 2024-03-25T19:19:26.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.469+0000 7f236ae63640 1 -- 172.21.15.6:0/507591245 >> 172.21.15.6:0/507591245 conn(0x7f23640617a0 msgr2=0x7f2364061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:19:26.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.469+0000 7f236ae63640 1 -- 172.21.15.6:0/507591245 shutdown_connections 2024-03-25T19:19:26.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.469+0000 7f236ae63640 1 -- 172.21.15.6:0/507591245 wait complete. 2024-03-25T19:19:26.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.470+0000 7f236ae63640 10 monclient: build_initial_monmap 2024-03-25T19:19:26.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.470+0000 7f236ae63640 10 monclient: monmap: 2024-03-25T19:19:26.471 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:19:26.471 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:19:26.471 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:19:26.472 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:19:26.472 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:26.472 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:19:26.472 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:19:26.472 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:19:26.472 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:19:26.472 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:26.472 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.470+0000 7f236ae63640 1 librados: starting msgr at 2024-03-25T19:19:26.472 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.470+0000 7f236ae63640 1 librados: starting objecter 2024-03-25T19:19:26.472 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.470+0000 7f236ae63640 1 Processor -- start 2024-03-25T19:19:26.472 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.470+0000 7f236ae63640 1 -- start start 2024-03-25T19:19:26.472 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.470+0000 7f236ae63640 1 librados: setting wanted keys 2024-03-25T19:19:26.473 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.470+0000 7f236ae63640 1 librados: calling monclient init 2024-03-25T19:19:26.473 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.470+0000 7f236ae63640 10 monclient: init 2024-03-25T19:19:26.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.471+0000 7f236ae63640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:26.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.471+0000 7f236ae63640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:19:26.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.471+0000 7f236ae63640 10 monclient(hunting): picked mon.noname-a con 0x7f23640b1b00 addr v1:172.21.15.6:6789/0 2024-03-25T19:19:26.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.471+0000 7f236ae63640 10 monclient(hunting): picked mon.noname-c con 0x7f23640b44b0 addr v1:172.21.15.6:6790/0 2024-03-25T19:19:26.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.471+0000 7f236ae63640 10 monclient(hunting): picked mon.noname-b con 0x7f23640b4e70 addr v1:172.21.15.192:6789/0 2024-03-25T19:19:26.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.471+0000 7f236ae63640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f236413a670 con 0x7f23640b1b00 2024-03-25T19:19:26.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.471+0000 7f236ae63640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2364134e50 con 0x7f23640b4e70 2024-03-25T19:19:26.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.471+0000 7f236ae63640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2364134fe0 con 0x7f23640b44b0 2024-03-25T19:19:26.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.471+0000 7f236ae63640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:26.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.471+0000 7f236ae63640 10 monclient(hunting): authenticate will time out at 6270.711460s 2024-03-25T19:19:26.491 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.492+0000 7f2368bff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f23640b1b00 0x7f2364131e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:42788/0 (socket says 172.21.15.6:42788) 2024-03-25T19:19:26.492 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.492+0000 7f2368bff640 1 -- 172.21.15.6:0/3003758093 learned_addr learned my addr 172.21.15.6:0/3003758093 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:19:26.492 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.492+0000 7f23617fa640 10 client.?.objecter ms_handle_connect 0x7f23640b1b00 2024-03-25T19:19:26.492 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.492+0000 7f23617fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:26.504 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.504+0000 7f23617fa640 10 client.?.objecter ms_handle_connect 0x7f23640b44b0 2024-03-25T19:19:26.519 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.504+0000 7f23617fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:26.520 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.510+0000 7f23617fa640 10 client.?.objecter ms_handle_connect 0x7f23640b4e70 2024-03-25T19:19:26.520 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.510+0000 7f23617fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:26.520 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.511+0000 7f23617fa640 1 -- 172.21.15.6:0/3003758093 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1709602897 0 0) 0x7f236413a670 con 0x7f23640b1b00 2024-03-25T19:19:26.520 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.511+0000 7f23617fa640 10 monclient(hunting): _negotiate 2024-03-25T19:19:26.520 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.511+0000 7f23617fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:26.520 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.511+0000 7f23617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:26.520 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.511+0000 7f23617fa640 10 monclient(hunting): my global_id is 4464 2024-03-25T19:19:26.520 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.511+0000 7f23617fa640 1 -- 172.21.15.6:0/3003758093 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2348004eb0 con 0x7f23640b1b00 2024-03-25T19:19:26.521 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.511+0000 7f23617fa640 1 -- 172.21.15.6:0/3003758093 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3149972029 0 0) 0x7f2364134fe0 con 0x7f23640b44b0 2024-03-25T19:19:26.521 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.511+0000 7f23617fa640 10 monclient(hunting): _negotiate 2024-03-25T19:19:26.521 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.511+0000 7f23617fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:26.521 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.511+0000 7f23617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:26.521 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.511+0000 7f23617fa640 10 monclient(hunting): my global_id is 4463 2024-03-25T19:19:26.521 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.511+0000 7f23617fa640 1 -- 172.21.15.6:0/3003758093 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f236413a670 con 0x7f23640b44b0 2024-03-25T19:19:26.526 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.526+0000 7f23617fa640 1 -- 172.21.15.6:0/3003758093 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2315384594 0 0) 0x7f2364134e50 con 0x7f23640b4e70 2024-03-25T19:19:26.526 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.527+0000 7f23617fa640 10 monclient(hunting): _negotiate 2024-03-25T19:19:26.527 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.527+0000 7f23617fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:26.527 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.527+0000 7f23617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:26.527 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.527+0000 7f23617fa640 10 monclient(hunting): my global_id is 4468 2024-03-25T19:19:26.527 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.527+0000 7f23617fa640 1 -- 172.21.15.6:0/3003758093 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2364134fe0 con 0x7f23640b4e70 2024-03-25T19:19:26.547 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.547+0000 7f23617fa640 1 -- 172.21.15.6:0/3003758093 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1793960789 0 0) 0x7f2364134fe0 con 0x7f23640b4e70 2024-03-25T19:19:26.548 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.547+0000 7f23617fa640 1 -- 172.21.15.6:0/3003758093 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2364134e50 con 0x7f23640b4e70 2024-03-25T19:19:26.548 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.547+0000 7f23617fa640 1 -- 172.21.15.6:0/3003758093 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f235c001cb0 con 0x7f23640b4e70 2024-03-25T19:19:26.548 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.547+0000 7f23617fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:26.548 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.547+0000 7f23617fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:19:26.548 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.547+0000 7f23617fa640 10 monclient(hunting): dump: 2024-03-25T19:19:26.548 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:19:26.548 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:26.548 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:26.548 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:26.548 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:26.548 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:26.548 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:19:26.548 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:19:26.549 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:19:26.549 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:26.549 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.547+0000 7f23617fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:19:26.549 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.549+0000 7f23617fa640 1 -- 172.21.15.6:0/3003758093 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1181894360 0 0) 0x7f236413a670 con 0x7f23640b44b0 2024-03-25T19:19:26.549 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.549+0000 7f23617fa640 1 -- 172.21.15.6:0/3003758093 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2364134fe0 con 0x7f23640b44b0 2024-03-25T19:19:26.549 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.549+0000 7f23617fa640 1 -- 172.21.15.6:0/3003758093 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f2350001b40 con 0x7f23640b44b0 2024-03-25T19:19:26.549 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.549+0000 7f23617fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:26.550 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.549+0000 7f23617fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:19:26.550 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.549+0000 7f23617fa640 10 monclient(hunting): dump: 2024-03-25T19:19:26.550 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:19:26.550 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:26.550 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:26.550 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:26.550 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:26.550 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:26.550 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:19:26.550 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:19:26.550 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:19:26.550 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:26.557 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.557+0000 7f23617fa640 1 -- 172.21.15.6:0/3003758093 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2480014638 0 0) 0x7f2348004eb0 con 0x7f23640b1b00 2024-03-25T19:19:26.557 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.557+0000 7f23617fa640 1 -- 172.21.15.6:0/3003758093 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f236413a670 con 0x7f23640b1b00 2024-03-25T19:19:26.557 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.557+0000 7f23617fa640 1 -- 172.21.15.6:0/3003758093 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f2358001930 con 0x7f23640b1b00 2024-03-25T19:19:26.557 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.557+0000 7f23617fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:26.557 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.557+0000 7f23617fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:19:26.557 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.557+0000 7f23617fa640 10 monclient(hunting): dump: 2024-03-25T19:19:26.557 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:19:26.557 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:26.557 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:26.557 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:26.558 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:26.558 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:26.558 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:19:26.558 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:19:26.558 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:19:26.558 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:26.571 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.571+0000 7f23617fa640 1 -- 172.21.15.6:0/3003758093 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1708791698 0 0) 0x7f2364134e50 con 0x7f23640b4e70 2024-03-25T19:19:26.572 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.571+0000 7f23617fa640 1 -- 172.21.15.6:0/3003758093 >> v1:172.21.15.6:6790/0 conn(0x7f23640b44b0 legacy=0x7f23641df370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:26.572 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.571+0000 7f23617fa640 1 -- 172.21.15.6:0/3003758093 >> v1:172.21.15.6:6789/0 conn(0x7f23640b1b00 legacy=0x7f2364131e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:26.572 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.571+0000 7f23617fa640 10 monclient: _finish_hunting 0 2024-03-25T19:19:26.572 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.571+0000 7f23617fa640 1 monclient: found mon.b 2024-03-25T19:19:26.572 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.571+0000 7f23617fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:19:26.572 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.571+0000 7f23617fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:19:26.572 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.571+0000 7f23617fa640 1 -- 172.21.15.6:0/3003758093 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2364135170 con 0x7f23640b4e70 2024-03-25T19:19:26.572 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.571+0000 7f23617fa640 10 monclient: _finish_auth 0 2024-03-25T19:19:26.572 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.571+0000 7f23617fa640 10 monclient: _check_auth_tickets 2024-03-25T19:19:26.572 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.571+0000 7f23617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:19:26.572 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.571+0000 7f236ae63640 5 monclient: authenticate success, global_id 4468 2024-03-25T19:19:26.572 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.571+0000 7f236ae63640 10 monclient: _renew_subs 2024-03-25T19:19:26.573 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.571+0000 7f236ae63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:19:26.573 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.571+0000 7f236ae63640 1 -- 172.21.15.6:0/3003758093 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2364132570 con 0x7f23640b4e70 2024-03-25T19:19:26.573 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.571+0000 7f236ae63640 10 client.4468.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:19:26.573 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.571+0000 7f236ae63640 10 monclient: _renew_subs 2024-03-25T19:19:26.573 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.571+0000 7f236ae63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:19:26.573 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.571+0000 7f236ae63640 1 -- 172.21.15.6:0/3003758093 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2364144e50 con 0x7f23640b4e70 2024-03-25T19:19:26.573 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.571+0000 7f236ae63640 1 librados: init done 2024-03-25T19:19:26.573 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.572+0000 7f236ae63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:26.574 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.572+0000 7f236ae63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:26.574 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.572+0000 7f236ae63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:19:26.574 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.572+0000 7f236ae63640 1 -- 172.21.15.6:0/3003758093 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f236404f730 con 0x7f23640b4e70 2024-03-25T19:19:26.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.841+0000 7f23617fa640 1 -- 172.21.15.6:0/3003758093 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f235c0020c0 con 0x7f23640b4e70 2024-03-25T19:19:26.842 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.841+0000 7f23617fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:19:26.842 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.841+0000 7f23617fa640 1 -- 172.21.15.6:0/3003758093 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f235c003b30 con 0x7f23640b4e70 2024-03-25T19:19:26.842 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.841+0000 7f23617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:26.842 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.841+0000 7f23617fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:19:26.842 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.841+0000 7f23617fa640 10 monclient: dump: 2024-03-25T19:19:26.842 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:19:26.842 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:26.842 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:26.842 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:26.842 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:26.842 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:26.842 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:19:26.843 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:19:26.843 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:19:26.843 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:26.843 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.841+0000 7f2362ffd640 4 set_mon_vals no callback set 2024-03-25T19:19:26.843 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.842+0000 7f23617fa640 1 -- 172.21.15.6:0/3003758093 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f235c019b40 con 0x7f23640b4e70 2024-03-25T19:19:26.843 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.843+0000 7f2368bff640 10 monclient: get_auth_request con 0x7f234806e0b0 auth_method 0 2024-03-25T19:19:26.844 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.843+0000 7f23617fa640 10 client.4468.objecter ms_handle_connect 0x7f234806e0b0 2024-03-25T19:19:26.849 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.849+0000 7f23617fa640 1 -- 172.21.15.6:0/3003758093 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f235c053840 con 0x7f23640b4e70 2024-03-25T19:19:26.849 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.849+0000 7f23617fa640 10 client.4468.objecter ms_dispatch 0x7f2364000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:19:26.849 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.849+0000 7f23617fa640 3 client.4468.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:19:26.849 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.849+0000 7f23617fa640 3 client.4468.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:19:26.849 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.849+0000 7f23617fa640 20 client.4468.objecter dump_active .. 0 homeless 2024-03-25T19:19:26.850 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.849+0000 7f23617fa640 1 -- 172.21.15.6:0/3003758093 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f235c04f180 con 0x7f23640b4e70 2024-03-25T19:19:26.850 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.849+0000 7f23617fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:26.850 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:26.849+0000 7f23617fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:19:27.079 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:27.079+0000 7f236ae63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:19:27.079 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:27.079+0000 7f236ae63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:19:27.080 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:27.079+0000 7f236ae63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:19:27.080 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:27.079+0000 7f236ae63640 1 -- 172.21.15.6:0/3003758093 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f236413a820 con 0x7f23640b4e70 2024-03-25T19:19:27.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:27.104+0000 7f23617fa640 1 -- 172.21.15.6:0/3003758093 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f236413a820 con 0x7f23640b4e70 2024-03-25T19:19:27.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:27.104+0000 7f23617fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:19:27.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:27.104+0000 7f23617fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:19:27.104 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:19:27.105 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:19:27.107 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:27.107+0000 7f236ae63640 10 librados: watch_flush enter 2024-03-25T19:19:27.107 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:27.107+0000 7f236ae63640 10 librados: watch_flush exit 2024-03-25T19:19:27.107 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:27.107+0000 7f236ae63640 20 client.4468.objecter shutdown clearing up homeless session... 2024-03-25T19:19:27.107 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:27.107+0000 7f236ae63640 10 client.4468.objecter successfully canceled tick 2024-03-25T19:19:27.108 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:27.107+0000 7f236ae63640 1 -- 172.21.15.6:0/3003758093 >> v1:172.21.15.6:6800/2936652650 conn(0x7f234806e0b0 legacy=0x7f2348070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:27.108 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:27.107+0000 7f236ae63640 10 monclient: shutdown 2024-03-25T19:19:27.108 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:27.107+0000 7f236ae63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:19:27.108 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:27.107+0000 7f236ae63640 1 -- 172.21.15.6:0/3003758093 >> v1:172.21.15.192:6789/0 conn(0x7f23640b4e70 legacy=0x7f23641dfa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:27.108 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:27.107+0000 7f236ae63640 1 -- 172.21.15.6:0/3003758093 shutdown_connections 2024-03-25T19:19:27.108 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:27.107+0000 7f236ae63640 1 -- 172.21.15.6:0/3003758093 >> 172.21.15.6:0/3003758093 conn(0x7f23640617a0 msgr2=0x7f236413ee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:19:27.108 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:27.107+0000 7f236ae63640 1 -- 172.21.15.6:0/3003758093 shutdown_connections 2024-03-25T19:19:27.108 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:27.107+0000 7f236ae63640 1 -- 172.21.15.6:0/3003758093 wait complete. 2024-03-25T19:19:27.108 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:27.107+0000 7f236ae63640 1 librados: shutdown 2024-03-25T19:19:27.118 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:19:33.924 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:19:34.038 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.038+0000 7f70f5a27640 10 monclient: get_monmap_and_config 2024-03-25T19:19:34.038 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.038+0000 7f70f5a27640 10 monclient: build_initial_monmap 2024-03-25T19:19:34.038 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.038+0000 7f70f5a27640 10 monclient: monmap: 2024-03-25T19:19:34.039 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:19:34.039 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:34.039 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:19:34.039678+0000 2024-03-25T19:19:34.039 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:19:34.039678+0000 2024-03-25T19:19:34.039 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:34.039 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:34.039 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:19:34.039 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:19:34.039 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:19:34.039 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:34.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.039+0000 7f70f5a27640 1 Processor -- start 2024-03-25T19:19:34.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.039+0000 7f70f5a27640 1 -- start start 2024-03-25T19:19:34.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.039+0000 7f70f5a27640 10 monclient: init 2024-03-25T19:19:34.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.040+0000 7f70f5a27640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:34.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.040+0000 7f70f5a27640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:19:34.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.040+0000 7f70f5a27640 10 monclient(hunting): picked mon.noname-c con 0x7f70f00b4ff0 addr v1:172.21.15.192:6789/0 2024-03-25T19:19:34.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.040+0000 7f70f5a27640 10 monclient(hunting): picked mon.noname-a con 0x7f70f00b4670 addr v1:172.21.15.6:6789/0 2024-03-25T19:19:34.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.040+0000 7f70f5a27640 10 monclient(hunting): picked mon.noname-b con 0x7f70f013c480 addr v1:172.21.15.6:6790/0 2024-03-25T19:19:34.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.040+0000 7f70f5a27640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70f00ab0e0 con 0x7f70f00b4670 2024-03-25T19:19:34.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.040+0000 7f70f5a27640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70f00ab3c0 con 0x7f70f00b4ff0 2024-03-25T19:19:34.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.040+0000 7f70f5a27640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70f00a9f30 con 0x7f70f013c480 2024-03-25T19:19:34.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.040+0000 7f70f5a27640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:34.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.040+0000 7f70f5a27640 10 monclient(hunting): authenticate will time out at 6278.280182s 2024-03-25T19:19:34.043 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.044+0000 7f70ee7fc640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f70f00b4670 0x7f70f01425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33748/0 (socket says 172.21.15.6:33748) 2024-03-25T19:19:34.044 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.044+0000 7f70ee7fc640 1 -- 172.21.15.6:0/351993512 learned_addr learned my addr 172.21.15.6:0/351993512 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:19:34.057 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.057+0000 7f70edffb640 1 -- 172.21.15.6:0/351993512 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3058108359 0 0) 0x7f70f00ab0e0 con 0x7f70f00b4670 2024-03-25T19:19:34.057 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.057+0000 7f70edffb640 10 monclient(hunting): _negotiate 2024-03-25T19:19:34.057 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.057+0000 7f70edffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:34.057 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.057+0000 7f70edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:34.057 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.057+0000 7f70edffb640 10 monclient(hunting): my global_id is 4467 2024-03-25T19:19:34.058 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.057+0000 7f70edffb640 1 -- 172.21.15.6:0/351993512 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70d8003520 con 0x7f70f00b4670 2024-03-25T19:19:34.064 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.064+0000 7f70edffb640 1 -- 172.21.15.6:0/351993512 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3867168937 0 0) 0x7f70d8003520 con 0x7f70f00b4670 2024-03-25T19:19:34.064 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.064+0000 7f70edffb640 1 -- 172.21.15.6:0/351993512 >> v1:172.21.15.6:6790/0 conn(0x7f70f013c480 legacy=0x7f70f014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:34.064 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.064+0000 7f70edffb640 1 -- 172.21.15.6:0/351993512 >> v1:172.21.15.192:6789/0 conn(0x7f70f00b4ff0 legacy=0x7f70f013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:34.064 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.064+0000 7f70edffb640 10 monclient: _finish_hunting 0 2024-03-25T19:19:34.064 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.064+0000 7f70edffb640 1 monclient: found mon.noname-a 2024-03-25T19:19:34.064 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.064+0000 7f70edffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:19:34.064 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.064+0000 7f70edffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-03-25T19:19:34.064 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.064+0000 7f70edffb640 1 -- 172.21.15.6:0/351993512 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70f00a9b70 con 0x7f70f00b4670 2024-03-25T19:19:34.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.064+0000 7f70edffb640 10 monclient: _finish_auth 0 2024-03-25T19:19:34.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.064+0000 7f70edffb640 10 monclient: _check_auth_tickets 2024-03-25T19:19:34.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.064+0000 7f70edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:19:34.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.064+0000 7f70edffb640 1 -- 172.21.15.6:0/351993512 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f70e4001690 con 0x7f70f00b4670 2024-03-25T19:19:34.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.064+0000 7f70edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:34.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.064+0000 7f70edffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:19:34.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.064+0000 7f70edffb640 10 monclient: dump: 2024-03-25T19:19:34.065 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:19:34.065 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:34.065 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:34.065 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:34.065 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:34.065 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:34.065 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:19:34.065 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:19:34.066 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:19:34.066 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:34.066 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.064+0000 7f70f5a27640 5 monclient: authenticate success, global_id 4467 2024-03-25T19:19:34.066 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.064+0000 7f70f5a27640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:19:34.070 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.070+0000 7f70edffb640 1 -- 172.21.15.6:0/351993512 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70e40017f0 con 0x7f70f00b4670 2024-03-25T19:19:34.070 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.070+0000 7f70edffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:19:34.070 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.070+0000 7f70edffb640 1 -- 172.21.15.6:0/351993512 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f70e4001c00 con 0x7f70f00b4670 2024-03-25T19:19:34.070 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.070+0000 7f70f5a27640 10 monclient: get_monmap_and_config success 2024-03-25T19:19:34.071 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.070+0000 7f70f5a27640 4 set_mon_vals no callback set 2024-03-25T19:19:34.071 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.070+0000 7f70edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:34.071 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.070+0000 7f70edffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:19:34.071 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.070+0000 7f70edffb640 10 monclient: dump: 2024-03-25T19:19:34.071 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:19:34.071 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:34.071 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:34.071 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:34.071 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:34.071 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:34.071 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:19:34.071 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:19:34.071 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:19:34.071 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:34.073 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.071+0000 7f70f5a27640 10 monclient: shutdown 2024-03-25T19:19:34.073 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.071+0000 7f70f5a27640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:19:34.073 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.071+0000 7f70f5a27640 1 -- 172.21.15.6:0/351993512 >> v1:172.21.15.6:6789/0 conn(0x7f70f00b4670 legacy=0x7f70f01425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:34.073 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.071+0000 7f70f5a27640 1 -- 172.21.15.6:0/351993512 shutdown_connections 2024-03-25T19:19:34.073 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.071+0000 7f70f5a27640 1 -- 172.21.15.6:0/351993512 >> 172.21.15.6:0/351993512 conn(0x7f70f00617a0 msgr2=0x7f70f0061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:19:34.073 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.071+0000 7f70f5a27640 1 -- 172.21.15.6:0/351993512 shutdown_connections 2024-03-25T19:19:34.073 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.071+0000 7f70f5a27640 1 -- 172.21.15.6:0/351993512 wait complete. 2024-03-25T19:19:34.073 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.072+0000 7f70f5a27640 10 monclient: build_initial_monmap 2024-03-25T19:19:34.073 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.072+0000 7f70f5a27640 10 monclient: monmap: 2024-03-25T19:19:34.073 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:19:34.074 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:19:34.074 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:19:34.074 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:19:34.074 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:34.074 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:19:34.074 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:19:34.074 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:19:34.074 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:19:34.074 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:34.074 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.072+0000 7f70f5a27640 1 librados: starting msgr at 2024-03-25T19:19:34.074 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.072+0000 7f70f5a27640 1 librados: starting objecter 2024-03-25T19:19:34.074 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.072+0000 7f70f5a27640 1 Processor -- start 2024-03-25T19:19:34.075 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.073+0000 7f70f5a27640 1 -- start start 2024-03-25T19:19:34.075 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.073+0000 7f70f5a27640 1 librados: setting wanted keys 2024-03-25T19:19:34.075 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.073+0000 7f70f5a27640 1 librados: calling monclient init 2024-03-25T19:19:34.075 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.073+0000 7f70f5a27640 10 monclient: init 2024-03-25T19:19:34.076 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.073+0000 7f70f5a27640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:34.076 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.073+0000 7f70f5a27640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:19:34.076 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.073+0000 7f70f5a27640 10 monclient(hunting): picked mon.noname-c con 0x7f70f00b4670 addr v1:172.21.15.6:6790/0 2024-03-25T19:19:34.076 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.073+0000 7f70f5a27640 10 monclient(hunting): picked mon.noname-b con 0x7f70f00b4ff0 addr v1:172.21.15.192:6789/0 2024-03-25T19:19:34.076 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.073+0000 7f70f5a27640 10 monclient(hunting): picked mon.noname-a con 0x7f70f013c480 addr v1:172.21.15.6:6789/0 2024-03-25T19:19:34.076 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.073+0000 7f70f5a27640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70f0143780 con 0x7f70f013c480 2024-03-25T19:19:34.076 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.073+0000 7f70f5a27640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70f0144da0 con 0x7f70f00b4ff0 2024-03-25T19:19:34.076 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.073+0000 7f70f5a27640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70f0145080 con 0x7f70f00b4670 2024-03-25T19:19:34.076 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.073+0000 7f70f5a27640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:34.076 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.073+0000 7f70f5a27640 10 monclient(hunting): authenticate will time out at 6278.313731s 2024-03-25T19:19:34.081 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.081+0000 7f70ee7fc640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f70f00b4ff0 0x7f70f0145af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:51450/0 (socket says 172.21.15.6:51450) 2024-03-25T19:19:34.081 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.081+0000 7f70ee7fc640 1 -- 172.21.15.6:0/1390656434 learned_addr learned my addr 172.21.15.6:0/1390656434 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:19:34.083 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.083+0000 7f70d7fff640 10 client.?.objecter ms_handle_connect 0x7f70f00b4670 2024-03-25T19:19:34.083 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.083+0000 7f70d7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:34.085 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.085+0000 7f70d7fff640 10 client.?.objecter ms_handle_connect 0x7f70f00b4ff0 2024-03-25T19:19:34.085 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.085+0000 7f70d7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:34.091 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.091+0000 7f70d7fff640 1 -- 172.21.15.6:0/1390656434 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3060112882 0 0) 0x7f70f0144da0 con 0x7f70f00b4ff0 2024-03-25T19:19:34.091 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.091+0000 7f70d7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:19:34.091 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.091+0000 7f70d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:34.091 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.091+0000 7f70d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:34.091 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.091+0000 7f70d7fff640 10 monclient(hunting): my global_id is 4474 2024-03-25T19:19:34.091 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.091+0000 7f70d7fff640 1 -- 172.21.15.6:0/1390656434 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70dc004eb0 con 0x7f70f00b4ff0 2024-03-25T19:19:34.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.099+0000 7f70d7fff640 1 -- 172.21.15.6:0/1390656434 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 943135032 0 0) 0x7f70f0145080 con 0x7f70f00b4670 2024-03-25T19:19:34.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.099+0000 7f70d7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:19:34.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.099+0000 7f70d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:34.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.099+0000 7f70d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:34.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.099+0000 7f70d7fff640 10 monclient(hunting): my global_id is 4469 2024-03-25T19:19:34.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.099+0000 7f70d7fff640 1 -- 172.21.15.6:0/1390656434 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70f0144da0 con 0x7f70f00b4670 2024-03-25T19:19:34.101 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.101+0000 7f70d7fff640 10 client.?.objecter ms_handle_connect 0x7f70f013c480 2024-03-25T19:19:34.101 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.101+0000 7f70d7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:34.110 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.110+0000 7f70d7fff640 1 -- 172.21.15.6:0/1390656434 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 970853451 0 0) 0x7f70dc004eb0 con 0x7f70f00b4ff0 2024-03-25T19:19:34.110 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.110+0000 7f70d7fff640 1 -- 172.21.15.6:0/1390656434 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70f0145080 con 0x7f70f00b4ff0 2024-03-25T19:19:34.111 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.110+0000 7f70d7fff640 1 -- 172.21.15.6:0/1390656434 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f70e4001c00 con 0x7f70f00b4ff0 2024-03-25T19:19:34.111 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.110+0000 7f70d7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:34.111 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.110+0000 7f70d7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:19:34.111 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.110+0000 7f70d7fff640 10 monclient(hunting): dump: 2024-03-25T19:19:34.111 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:19:34.111 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:34.111 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:34.111 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:34.111 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:34.111 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:34.111 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:19:34.111 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:19:34.111 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:19:34.111 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:34.111 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.110+0000 7f70d7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:19:34.115 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.115+0000 7f70d7fff640 1 -- 172.21.15.6:0/1390656434 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 967809524 0 0) 0x7f70f0143780 con 0x7f70f013c480 2024-03-25T19:19:34.115 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.115+0000 7f70d7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:19:34.115 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.115+0000 7f70d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:34.115 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.115+0000 7f70d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:34.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.115+0000 7f70d7fff640 10 monclient(hunting): my global_id is 4470 2024-03-25T19:19:34.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.115+0000 7f70d7fff640 1 -- 172.21.15.6:0/1390656434 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70dc004eb0 con 0x7f70f013c480 2024-03-25T19:19:34.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.119+0000 7f70d7fff640 1 -- 172.21.15.6:0/1390656434 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1731339474 0 0) 0x7f70f0144da0 con 0x7f70f00b4670 2024-03-25T19:19:34.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.119+0000 7f70d7fff640 1 -- 172.21.15.6:0/1390656434 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70f0143780 con 0x7f70f00b4670 2024-03-25T19:19:34.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.119+0000 7f70d7fff640 1 -- 172.21.15.6:0/1390656434 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f70e00018a0 con 0x7f70f00b4670 2024-03-25T19:19:34.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.119+0000 7f70d7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:34.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.119+0000 7f70d7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:19:34.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.119+0000 7f70d7fff640 10 monclient(hunting): dump: 2024-03-25T19:19:34.119 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:19:34.119 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:34.119 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:34.119 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:34.120 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:34.120 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:34.120 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:19:34.120 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:19:34.120 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:19:34.120 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:34.123 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.123+0000 7f70d7fff640 1 -- 172.21.15.6:0/1390656434 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3127444862 0 0) 0x7f70f0145080 con 0x7f70f00b4ff0 2024-03-25T19:19:34.123 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.123+0000 7f70d7fff640 1 -- 172.21.15.6:0/1390656434 >> v1:172.21.15.6:6790/0 conn(0x7f70f00b4670 legacy=0x7f70f013dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:34.123 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.123+0000 7f70d7fff640 1 -- 172.21.15.6:0/1390656434 >> v1:172.21.15.6:6789/0 conn(0x7f70f013c480 legacy=0x7f70f01e2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:34.123 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.123+0000 7f70d7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:19:34.123 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.123+0000 7f70d7fff640 1 monclient: found mon.b 2024-03-25T19:19:34.123 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.123+0000 7f70d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:19:34.123 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.123+0000 7f70d7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:19:34.123 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.123+0000 7f70d7fff640 1 -- 172.21.15.6:0/1390656434 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70f013f2e0 con 0x7f70f00b4ff0 2024-03-25T19:19:34.124 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.123+0000 7f70d7fff640 10 monclient: _finish_auth 0 2024-03-25T19:19:34.124 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.123+0000 7f70d7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:19:34.124 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.123+0000 7f70d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:19:34.124 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.123+0000 7f70f5a27640 5 monclient: authenticate success, global_id 4474 2024-03-25T19:19:34.124 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.123+0000 7f70f5a27640 10 monclient: _renew_subs 2024-03-25T19:19:34.124 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.123+0000 7f70f5a27640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:19:34.124 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.123+0000 7f70f5a27640 1 -- 172.21.15.6:0/1390656434 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70f013f790 con 0x7f70f00b4ff0 2024-03-25T19:19:34.124 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.123+0000 7f70f5a27640 10 client.4474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:19:34.124 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.123+0000 7f70f5a27640 10 monclient: _renew_subs 2024-03-25T19:19:34.124 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.123+0000 7f70f5a27640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:19:34.124 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.123+0000 7f70f5a27640 1 -- 172.21.15.6:0/1390656434 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f70f01463e0 con 0x7f70f00b4ff0 2024-03-25T19:19:34.124 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.123+0000 7f70f5a27640 1 librados: init done 2024-03-25T19:19:34.125 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.124+0000 7f70f5a27640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:34.125 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.124+0000 7f70f5a27640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:34.125 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.124+0000 7f70f5a27640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:19:34.125 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.124+0000 7f70f5a27640 1 -- 172.21.15.6:0/1390656434 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70f01455a0 con 0x7f70f00b4ff0 2024-03-25T19:19:34.400 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.400+0000 7f70d7fff640 1 -- 172.21.15.6:0/1390656434 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70e4004a90 con 0x7f70f00b4ff0 2024-03-25T19:19:34.400 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.400+0000 7f70d7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:19:34.400 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.400+0000 7f70d7fff640 1 -- 172.21.15.6:0/1390656434 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f70e4004c80 con 0x7f70f00b4ff0 2024-03-25T19:19:34.400 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.400+0000 7f70d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:34.400 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.400+0000 7f70d7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:19:34.401 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.400+0000 7f70d7fff640 10 monclient: dump: 2024-03-25T19:19:34.401 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:19:34.401 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:34.401 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:34.401 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:34.401 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:34.401 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:34.401 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:19:34.401 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:19:34.401 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:19:34.401 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:34.401 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.400+0000 7f70ed7fa640 4 set_mon_vals no callback set 2024-03-25T19:19:34.402 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.401+0000 7f70d7fff640 1 -- 172.21.15.6:0/1390656434 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f70e4004ea0 con 0x7f70f00b4ff0 2024-03-25T19:19:34.402 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.401+0000 7f70eeffd640 10 monclient: get_auth_request con 0x7f70dc06dfb0 auth_method 0 2024-03-25T19:19:34.402 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.402+0000 7f70d7fff640 10 client.4474.objecter ms_handle_connect 0x7f70dc06dfb0 2024-03-25T19:19:34.406 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.406+0000 7f70d7fff640 1 -- 172.21.15.6:0/1390656434 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f70e40547b0 con 0x7f70f00b4ff0 2024-03-25T19:19:34.407 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.406+0000 7f70d7fff640 10 client.4474.objecter ms_dispatch 0x7f70f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:19:34.407 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.406+0000 7f70d7fff640 3 client.4474.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:19:34.407 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.406+0000 7f70d7fff640 3 client.4474.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:19:34.407 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.406+0000 7f70d7fff640 20 client.4474.objecter dump_active .. 0 homeless 2024-03-25T19:19:34.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.408+0000 7f70d7fff640 1 -- 172.21.15.6:0/1390656434 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f70e40502c0 con 0x7f70f00b4ff0 2024-03-25T19:19:34.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.408+0000 7f70d7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:34.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.408+0000 7f70d7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:19:34.652 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.652+0000 7f70f5a27640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:19:34.652 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.652+0000 7f70f5a27640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:19:34.652 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.652+0000 7f70f5a27640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:19:34.652 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.652+0000 7f70f5a27640 1 -- 172.21.15.6:0/1390656434 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70f013f470 con 0x7f70f00b4ff0 2024-03-25T19:19:34.681 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.681+0000 7f70d7fff640 1 -- 172.21.15.6:0/1390656434 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f70e40521f0 con 0x7f70f00b4ff0 2024-03-25T19:19:34.681 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.681+0000 7f70d7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:19:34.681 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.681+0000 7f70d7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:19:34.681 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:19:34.681 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:19:34.684 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.683+0000 7f70f5a27640 10 librados: watch_flush enter 2024-03-25T19:19:34.684 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.683+0000 7f70f5a27640 10 librados: watch_flush exit 2024-03-25T19:19:34.684 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.683+0000 7f70f5a27640 20 client.4474.objecter shutdown clearing up homeless session... 2024-03-25T19:19:34.684 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.683+0000 7f70f5a27640 10 client.4474.objecter successfully canceled tick 2024-03-25T19:19:34.685 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.683+0000 7f70f5a27640 1 -- 172.21.15.6:0/1390656434 >> v1:172.21.15.6:6800/2936652650 conn(0x7f70dc06dfb0 legacy=0x7f70dc070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:34.685 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.684+0000 7f70f5a27640 10 monclient: shutdown 2024-03-25T19:19:34.685 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.684+0000 7f70f5a27640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:19:34.685 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.684+0000 7f70f5a27640 1 -- 172.21.15.6:0/1390656434 >> v1:172.21.15.192:6789/0 conn(0x7f70f00b4ff0 legacy=0x7f70f0145af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:34.685 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.684+0000 7f70f5a27640 1 -- 172.21.15.6:0/1390656434 shutdown_connections 2024-03-25T19:19:34.685 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.684+0000 7f70f5a27640 1 -- 172.21.15.6:0/1390656434 >> 172.21.15.6:0/1390656434 conn(0x7f70f00617a0 msgr2=0x7f70f0140670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:19:34.685 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.684+0000 7f70f5a27640 1 -- 172.21.15.6:0/1390656434 shutdown_connections 2024-03-25T19:19:34.685 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.684+0000 7f70f5a27640 1 -- 172.21.15.6:0/1390656434 wait complete. 2024-03-25T19:19:34.685 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:34.684+0000 7f70f5a27640 1 librados: shutdown 2024-03-25T19:19:34.696 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:19:41.503 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:19:41.622 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.621+0000 7efd6c458640 10 monclient: get_monmap_and_config 2024-03-25T19:19:41.622 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.621+0000 7efd6c458640 10 monclient: build_initial_monmap 2024-03-25T19:19:41.622 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.622+0000 7efd6c458640 10 monclient: monmap: 2024-03-25T19:19:41.622 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:19:41.622 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:41.622 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:19:41.622980+0000 2024-03-25T19:19:41.622 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:19:41.622980+0000 2024-03-25T19:19:41.623 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:41.623 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:41.623 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:19:41.623 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:19:41.623 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:19:41.623 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:41.624 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.622+0000 7efd6c458640 1 Processor -- start 2024-03-25T19:19:41.624 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.622+0000 7efd6c458640 1 -- start start 2024-03-25T19:19:41.624 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.622+0000 7efd6c458640 10 monclient: init 2024-03-25T19:19:41.624 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.623+0000 7efd6c458640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:41.624 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.623+0000 7efd6c458640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:19:41.624 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.623+0000 7efd6c458640 10 monclient(hunting): picked mon.noname-a con 0x7efd640b4ff0 addr v1:172.21.15.6:6789/0 2024-03-25T19:19:41.624 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.624+0000 7efd6c458640 10 monclient(hunting): picked mon.noname-c con 0x7efd640b4670 addr v1:172.21.15.192:6789/0 2024-03-25T19:19:41.624 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.624+0000 7efd6c458640 10 monclient(hunting): picked mon.noname-b con 0x7efd6413c480 addr v1:172.21.15.6:6790/0 2024-03-25T19:19:41.624 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.624+0000 7efd6c458640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd640ab0e0 con 0x7efd640b4ff0 2024-03-25T19:19:41.624 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.624+0000 7efd6c458640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd640ab3c0 con 0x7efd640b4670 2024-03-25T19:19:41.625 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.624+0000 7efd6c458640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd640a9f30 con 0x7efd6413c480 2024-03-25T19:19:41.625 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.624+0000 7efd6c458640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:41.625 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.624+0000 7efd6c458640 10 monclient(hunting): authenticate will time out at 6285.864021s 2024-03-25T19:19:41.629 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.629+0000 7efd699f3640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7efd640b4670 0x7efd641425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:57774/0 (socket says 172.21.15.6:57774) 2024-03-25T19:19:41.629 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.629+0000 7efd699f3640 1 -- 172.21.15.6:0/2711393242 learned_addr learned my addr 172.21.15.6:0/2711393242 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:19:41.643 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.643+0000 7efd691f2640 1 -- 172.21.15.6:0/2711393242 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2997412008 0 0) 0x7efd640ab3c0 con 0x7efd640b4670 2024-03-25T19:19:41.643 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.643+0000 7efd691f2640 10 monclient(hunting): _negotiate 2024-03-25T19:19:41.644 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.643+0000 7efd691f2640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:41.644 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.643+0000 7efd691f2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:41.644 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.643+0000 7efd691f2640 10 monclient(hunting): my global_id is 4477 2024-03-25T19:19:41.644 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.643+0000 7efd691f2640 1 -- 172.21.15.6:0/2711393242 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd50003520 con 0x7efd640b4670 2024-03-25T19:19:41.651 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.652+0000 7efd691f2640 1 -- 172.21.15.6:0/2711393242 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3962201364 0 0) 0x7efd640a9f30 con 0x7efd6413c480 2024-03-25T19:19:41.652 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.652+0000 7efd691f2640 10 monclient(hunting): _negotiate 2024-03-25T19:19:41.652 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.652+0000 7efd691f2640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:41.652 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.652+0000 7efd691f2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:41.652 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.652+0000 7efd691f2640 10 monclient(hunting): my global_id is 4472 2024-03-25T19:19:41.652 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.652+0000 7efd691f2640 1 -- 172.21.15.6:0/2711393242 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd640ab3c0 con 0x7efd6413c480 2024-03-25T19:19:41.667 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.667+0000 7efd691f2640 1 -- 172.21.15.6:0/2711393242 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4103807413 0 0) 0x7efd640ab0e0 con 0x7efd640b4ff0 2024-03-25T19:19:41.667 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.667+0000 7efd691f2640 10 monclient(hunting): _negotiate 2024-03-25T19:19:41.667 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.667+0000 7efd691f2640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:41.667 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.667+0000 7efd691f2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:41.667 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.667+0000 7efd691f2640 10 monclient(hunting): my global_id is 4473 2024-03-25T19:19:41.668 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.667+0000 7efd691f2640 1 -- 172.21.15.6:0/2711393242 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd640a9f30 con 0x7efd640b4ff0 2024-03-25T19:19:41.675 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.675+0000 7efd691f2640 1 -- 172.21.15.6:0/2711393242 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2821381613 0 0) 0x7efd50003520 con 0x7efd640b4670 2024-03-25T19:19:41.675 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.675+0000 7efd691f2640 1 -- 172.21.15.6:0/2711393242 >> v1:172.21.15.6:6790/0 conn(0x7efd6413c480 legacy=0x7efd6414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:41.676 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.675+0000 7efd691f2640 1 -- 172.21.15.6:0/2711393242 >> v1:172.21.15.6:6789/0 conn(0x7efd640b4ff0 legacy=0x7efd6413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:41.676 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.675+0000 7efd691f2640 10 monclient: _finish_hunting 0 2024-03-25T19:19:41.676 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.675+0000 7efd691f2640 1 monclient: found mon.noname-c 2024-03-25T19:19:41.676 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.675+0000 7efd691f2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:19:41.676 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.675+0000 7efd691f2640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:19:41.676 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.675+0000 7efd691f2640 1 -- 172.21.15.6:0/2711393242 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd640a9b70 con 0x7efd640b4670 2024-03-25T19:19:41.676 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.675+0000 7efd691f2640 10 monclient: _finish_auth 0 2024-03-25T19:19:41.676 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.675+0000 7efd691f2640 10 monclient: _check_auth_tickets 2024-03-25T19:19:41.676 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.675+0000 7efd691f2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:19:41.676 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.675+0000 7efd691f2640 1 -- 172.21.15.6:0/2711393242 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7efd54001690 con 0x7efd640b4670 2024-03-25T19:19:41.676 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.675+0000 7efd691f2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:41.676 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.675+0000 7efd691f2640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:19:41.676 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.675+0000 7efd691f2640 10 monclient: dump: 2024-03-25T19:19:41.676 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:19:41.677 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:41.677 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:41.677 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:41.677 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:41.677 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:41.677 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:19:41.677 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:19:41.677 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:19:41.677 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:41.677 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.675+0000 7efd6c458640 5 monclient: authenticate success, global_id 4477 2024-03-25T19:19:41.677 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.675+0000 7efd6c458640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:19:41.689 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.689+0000 7efd691f2640 1 -- 172.21.15.6:0/2711393242 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd540017f0 con 0x7efd640b4670 2024-03-25T19:19:41.689 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.689+0000 7efd691f2640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:19:41.689 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.689+0000 7efd691f2640 1 -- 172.21.15.6:0/2711393242 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7efd54001c00 con 0x7efd640b4670 2024-03-25T19:19:41.689 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.689+0000 7efd6c458640 10 monclient: get_monmap_and_config success 2024-03-25T19:19:41.689 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.689+0000 7efd6c458640 4 set_mon_vals no callback set 2024-03-25T19:19:41.689 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.689+0000 7efd691f2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:41.689 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.689+0000 7efd691f2640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:19:41.689 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.689+0000 7efd691f2640 10 monclient: dump: 2024-03-25T19:19:41.690 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:19:41.690 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:41.690 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:41.690 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:41.690 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:41.690 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:41.690 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:19:41.690 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:19:41.690 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:19:41.690 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:41.692 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.690+0000 7efd6c458640 10 monclient: shutdown 2024-03-25T19:19:41.692 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.690+0000 7efd6c458640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:19:41.692 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.690+0000 7efd6c458640 1 -- 172.21.15.6:0/2711393242 >> v1:172.21.15.192:6789/0 conn(0x7efd640b4670 legacy=0x7efd641425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:41.692 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.690+0000 7efd6c458640 1 -- 172.21.15.6:0/2711393242 shutdown_connections 2024-03-25T19:19:41.692 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.690+0000 7efd6c458640 1 -- 172.21.15.6:0/2711393242 >> 172.21.15.6:0/2711393242 conn(0x7efd640617a0 msgr2=0x7efd64061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:19:41.692 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.690+0000 7efd6c458640 1 -- 172.21.15.6:0/2711393242 shutdown_connections 2024-03-25T19:19:41.692 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.690+0000 7efd6c458640 1 -- 172.21.15.6:0/2711393242 wait complete. 2024-03-25T19:19:41.692 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.691+0000 7efd6c458640 10 monclient: build_initial_monmap 2024-03-25T19:19:41.693 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.691+0000 7efd6c458640 10 monclient: monmap: 2024-03-25T19:19:41.693 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:19:41.693 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:19:41.693 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:19:41.693 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:19:41.693 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:41.693 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:19:41.693 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:19:41.693 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:19:41.693 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:19:41.693 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:41.693 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.691+0000 7efd6c458640 1 librados: starting msgr at 2024-03-25T19:19:41.693 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.691+0000 7efd6c458640 1 librados: starting objecter 2024-03-25T19:19:41.693 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.691+0000 7efd6c458640 1 Processor -- start 2024-03-25T19:19:41.694 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.691+0000 7efd6c458640 1 -- start start 2024-03-25T19:19:41.694 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.691+0000 7efd6c458640 1 librados: setting wanted keys 2024-03-25T19:19:41.694 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.691+0000 7efd6c458640 1 librados: calling monclient init 2024-03-25T19:19:41.694 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.691+0000 7efd6c458640 10 monclient: init 2024-03-25T19:19:41.694 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.692+0000 7efd6c458640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:41.694 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.692+0000 7efd6c458640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:19:41.694 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.692+0000 7efd6c458640 10 monclient(hunting): picked mon.noname-c con 0x7efd640b4670 addr v1:172.21.15.6:6790/0 2024-03-25T19:19:41.694 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.692+0000 7efd6c458640 10 monclient(hunting): picked mon.noname-a con 0x7efd640b4ff0 addr v1:172.21.15.6:6789/0 2024-03-25T19:19:41.694 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.692+0000 7efd6c458640 10 monclient(hunting): picked mon.noname-b con 0x7efd6413c480 addr v1:172.21.15.192:6789/0 2024-03-25T19:19:41.694 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.692+0000 7efd6c458640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd64143810 con 0x7efd640b4ff0 2024-03-25T19:19:41.694 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.692+0000 7efd6c458640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd64130ad0 con 0x7efd6413c480 2024-03-25T19:19:41.694 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.692+0000 7efd6c458640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd64144120 con 0x7efd640b4670 2024-03-25T19:19:41.695 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.692+0000 7efd6c458640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:41.695 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.692+0000 7efd6c458640 10 monclient(hunting): authenticate will time out at 6285.932030s 2024-03-25T19:19:41.705 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.705+0000 7efd6a9f5640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7efd6413c480 0x7efd641303c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:57838/0 (socket says 172.21.15.6:57838) 2024-03-25T19:19:41.705 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.705+0000 7efd6a9f5640 1 -- 172.21.15.6:0/2044468295 learned_addr learned my addr 172.21.15.6:0/2044468295 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:19:41.709 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.709+0000 7efd5affd640 10 client.?.objecter ms_handle_connect 0x7efd6413c480 2024-03-25T19:19:41.709 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.709+0000 7efd5affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:41.717 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.717+0000 7efd5affd640 10 client.?.objecter ms_handle_connect 0x7efd640b4670 2024-03-25T19:19:41.717 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.717+0000 7efd5affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:41.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.721+0000 7efd5affd640 1 -- 172.21.15.6:0/2044468295 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2584130376 0 0) 0x7efd64130ad0 con 0x7efd6413c480 2024-03-25T19:19:41.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.721+0000 7efd5affd640 10 monclient(hunting): _negotiate 2024-03-25T19:19:41.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.721+0000 7efd5affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:41.722 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.721+0000 7efd5affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:41.722 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.721+0000 7efd5affd640 10 monclient(hunting): my global_id is 4480 2024-03-25T19:19:41.722 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.721+0000 7efd5affd640 1 -- 172.21.15.6:0/2044468295 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd48004eb0 con 0x7efd6413c480 2024-03-25T19:19:41.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.723+0000 7efd5affd640 10 client.?.objecter ms_handle_connect 0x7efd640b4ff0 2024-03-25T19:19:41.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.723+0000 7efd5affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:41.736 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.736+0000 7efd5affd640 1 -- 172.21.15.6:0/2044468295 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3335336158 0 0) 0x7efd64144120 con 0x7efd640b4670 2024-03-25T19:19:41.736 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.736+0000 7efd5affd640 10 monclient(hunting): _negotiate 2024-03-25T19:19:41.736 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.736+0000 7efd5affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:41.736 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.736+0000 7efd5affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:41.736 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.736+0000 7efd5affd640 10 monclient(hunting): my global_id is 4475 2024-03-25T19:19:41.736 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.736+0000 7efd5affd640 1 -- 172.21.15.6:0/2044468295 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd64130ad0 con 0x7efd640b4670 2024-03-25T19:19:41.739 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.739+0000 7efd5affd640 1 -- 172.21.15.6:0/2044468295 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 397950204 0 0) 0x7efd64143810 con 0x7efd640b4ff0 2024-03-25T19:19:41.739 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.739+0000 7efd5affd640 10 monclient(hunting): _negotiate 2024-03-25T19:19:41.739 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.739+0000 7efd5affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:41.739 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.739+0000 7efd5affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:41.739 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.739+0000 7efd5affd640 10 monclient(hunting): my global_id is 4476 2024-03-25T19:19:41.739 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.739+0000 7efd5affd640 1 -- 172.21.15.6:0/2044468295 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd64144120 con 0x7efd640b4ff0 2024-03-25T19:19:41.741 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.741+0000 7efd5affd640 1 -- 172.21.15.6:0/2044468295 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3662995406 0 0) 0x7efd48004eb0 con 0x7efd6413c480 2024-03-25T19:19:41.741 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.741+0000 7efd5affd640 1 -- 172.21.15.6:0/2044468295 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd64143810 con 0x7efd6413c480 2024-03-25T19:19:41.741 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.741+0000 7efd5affd640 1 -- 172.21.15.6:0/2044468295 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7efd5c001c10 con 0x7efd6413c480 2024-03-25T19:19:41.741 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.741+0000 7efd5affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:41.741 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.741+0000 7efd5affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:19:41.741 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.741+0000 7efd5affd640 10 monclient(hunting): dump: 2024-03-25T19:19:41.741 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:19:41.741 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:41.741 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:41.741 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:41.741 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:41.742 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:41.742 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:19:41.742 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:19:41.742 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:19:41.742 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:41.742 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.741+0000 7efd5affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:19:41.750 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.750+0000 7efd5affd640 1 -- 172.21.15.6:0/2044468295 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2786141917 0 0) 0x7efd64130ad0 con 0x7efd640b4670 2024-03-25T19:19:41.750 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.750+0000 7efd5affd640 1 -- 172.21.15.6:0/2044468295 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd48004eb0 con 0x7efd640b4670 2024-03-25T19:19:41.751 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.750+0000 7efd5affd640 1 -- 172.21.15.6:0/2044468295 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7efd600018f0 con 0x7efd640b4670 2024-03-25T19:19:41.751 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.750+0000 7efd5affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:41.751 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.750+0000 7efd5affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:19:41.751 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.750+0000 7efd5affd640 10 monclient(hunting): dump: 2024-03-25T19:19:41.751 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:19:41.751 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:41.751 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:41.751 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:41.751 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:41.751 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:41.751 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:19:41.751 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:19:41.751 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:19:41.751 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:41.753 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.752+0000 7efd5affd640 1 -- 172.21.15.6:0/2044468295 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1982631717 0 0) 0x7efd64143810 con 0x7efd6413c480 2024-03-25T19:19:41.753 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.752+0000 7efd5affd640 1 -- 172.21.15.6:0/2044468295 >> v1:172.21.15.6:6790/0 conn(0x7efd640b4670 legacy=0x7efd64144de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:41.753 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.752+0000 7efd5affd640 1 -- 172.21.15.6:0/2044468295 >> v1:172.21.15.6:6789/0 conn(0x7efd640b4ff0 legacy=0x7efd6412fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:41.753 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.752+0000 7efd5affd640 10 monclient: _finish_hunting 0 2024-03-25T19:19:41.753 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.752+0000 7efd5affd640 1 monclient: found mon.b 2024-03-25T19:19:41.753 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.752+0000 7efd5affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:19:41.753 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.752+0000 7efd5affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:19:41.753 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.752+0000 7efd5affd640 1 -- 172.21.15.6:0/2044468295 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd64144400 con 0x7efd6413c480 2024-03-25T19:19:41.753 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.752+0000 7efd5affd640 10 monclient: _finish_auth 0 2024-03-25T19:19:41.753 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.752+0000 7efd5affd640 10 monclient: _check_auth_tickets 2024-03-25T19:19:41.753 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.752+0000 7efd5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:19:41.753 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.752+0000 7efd6c458640 5 monclient: authenticate success, global_id 4480 2024-03-25T19:19:41.753 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.752+0000 7efd6c458640 10 monclient: _renew_subs 2024-03-25T19:19:41.753 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.752+0000 7efd6c458640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:19:41.753 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.752+0000 7efd6c458640 1 -- 172.21.15.6:0/2044468295 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd641446e0 con 0x7efd6413c480 2024-03-25T19:19:41.754 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.752+0000 7efd6c458640 10 client.4480.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:19:41.754 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.752+0000 7efd6c458640 10 monclient: _renew_subs 2024-03-25T19:19:41.754 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.752+0000 7efd6c458640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:19:41.754 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.752+0000 7efd6c458640 1 -- 172.21.15.6:0/2044468295 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efd641454f0 con 0x7efd6413c480 2024-03-25T19:19:41.754 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.752+0000 7efd6c458640 1 librados: init done 2024-03-25T19:19:41.754 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.753+0000 7efd6c458640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:41.754 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.753+0000 7efd6c458640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:41.754 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.753+0000 7efd6c458640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:19:41.754 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:41.753+0000 7efd6c458640 1 -- 172.21.15.6:0/2044468295 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd6406dbd0 con 0x7efd6413c480 2024-03-25T19:19:42.048 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:42.048+0000 7efd5affd640 1 -- 172.21.15.6:0/2044468295 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd5c003990 con 0x7efd6413c480 2024-03-25T19:19:42.049 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:42.048+0000 7efd5affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:19:42.049 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:42.048+0000 7efd5affd640 1 -- 172.21.15.6:0/2044468295 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7efd5c003da0 con 0x7efd6413c480 2024-03-25T19:19:42.049 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:42.048+0000 7efd5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:42.049 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:42.048+0000 7efd5affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:19:42.049 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:42.048+0000 7efd5affd640 10 monclient: dump: 2024-03-25T19:19:42.049 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:19:42.049 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:42.049 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:42.049 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:42.049 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:42.049 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:42.050 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:19:42.050 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:19:42.050 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:19:42.050 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:42.050 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:42.048+0000 7efd689f1640 4 set_mon_vals no callback set 2024-03-25T19:19:42.050 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:42.049+0000 7efd5affd640 1 -- 172.21.15.6:0/2044468295 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7efd5c003fc0 con 0x7efd6413c480 2024-03-25T19:19:42.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:42.049+0000 7efd6a1f4640 10 monclient: get_auth_request con 0x7efd4806df60 auth_method 0 2024-03-25T19:19:42.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:42.050+0000 7efd5affd640 10 client.4480.objecter ms_handle_connect 0x7efd4806df60 2024-03-25T19:19:42.057 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:42.057+0000 7efd5affd640 1 -- 172.21.15.6:0/2044468295 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7efd5c054a20 con 0x7efd6413c480 2024-03-25T19:19:42.057 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:42.057+0000 7efd5affd640 10 client.4480.objecter ms_dispatch 0x7efd64000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:19:42.057 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:42.057+0000 7efd5affd640 3 client.4480.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:19:42.057 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:42.057+0000 7efd5affd640 3 client.4480.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:19:42.057 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:42.057+0000 7efd5affd640 20 client.4480.objecter dump_active .. 0 homeless 2024-03-25T19:19:42.058 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:42.057+0000 7efd5affd640 1 -- 172.21.15.6:0/2044468295 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efd5c050530 con 0x7efd6413c480 2024-03-25T19:19:42.058 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:42.057+0000 7efd5affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:42.058 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:42.057+0000 7efd5affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:19:42.284 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:42.284+0000 7efd6c458640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:19:42.284 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:42.284+0000 7efd6c458640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:19:42.284 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:42.284+0000 7efd6c458640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:19:42.284 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:42.284+0000 7efd6c458640 1 -- 172.21.15.6:0/2044468295 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efd640688c0 con 0x7efd6413c480 2024-03-25T19:19:42.365 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:42.365+0000 7efd5affd640 1 -- 172.21.15.6:0/2044468295 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7efd5c052460 con 0x7efd6413c480 2024-03-25T19:19:42.366 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:42.365+0000 7efd5affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:19:42.366 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:42.365+0000 7efd5affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:19:42.366 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:19:42.366 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:19:42.368 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:42.368+0000 7efd6c458640 10 librados: watch_flush enter 2024-03-25T19:19:42.368 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:42.368+0000 7efd6c458640 10 librados: watch_flush exit 2024-03-25T19:19:42.368 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:42.368+0000 7efd6c458640 20 client.4480.objecter shutdown clearing up homeless session... 2024-03-25T19:19:42.368 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:42.368+0000 7efd6c458640 10 client.4480.objecter successfully canceled tick 2024-03-25T19:19:42.368 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:42.368+0000 7efd6c458640 1 -- 172.21.15.6:0/2044468295 >> v1:172.21.15.6:6800/2936652650 conn(0x7efd4806df60 legacy=0x7efd48070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:42.369 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:42.368+0000 7efd6c458640 10 monclient: shutdown 2024-03-25T19:19:42.369 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:42.368+0000 7efd6c458640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:19:42.369 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:42.368+0000 7efd6c458640 1 -- 172.21.15.6:0/2044468295 >> v1:172.21.15.192:6789/0 conn(0x7efd6413c480 legacy=0x7efd641303c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:42.369 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:42.368+0000 7efd6c458640 1 -- 172.21.15.6:0/2044468295 shutdown_connections 2024-03-25T19:19:42.369 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:42.368+0000 7efd6c458640 1 -- 172.21.15.6:0/2044468295 >> 172.21.15.6:0/2044468295 conn(0x7efd640617a0 msgr2=0x7efd64140760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:19:42.369 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:42.368+0000 7efd6c458640 1 -- 172.21.15.6:0/2044468295 shutdown_connections 2024-03-25T19:19:42.369 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:42.368+0000 7efd6c458640 1 -- 172.21.15.6:0/2044468295 wait complete. 2024-03-25T19:19:42.369 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:42.368+0000 7efd6c458640 1 librados: shutdown 2024-03-25T19:19:42.380 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:19:43.575 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:06:00.922 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:19:43.589 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:06:00.936 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:19:43.619 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:06:00.966 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:19:43.627 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:06:00.945 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:19:43.631 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:06:00.949 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:19:43.637 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:06:00.983 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:19:43.650 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:06:00.997 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:19:43.665 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:06:00.983 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:19:43.699 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:06:01.045 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:19:49.187 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:19:49.306 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.306+0000 7f90896ba640 10 monclient: get_monmap_and_config 2024-03-25T19:19:49.306 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.306+0000 7f90896ba640 10 monclient: build_initial_monmap 2024-03-25T19:19:49.307 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.306+0000 7f90896ba640 10 monclient: monmap: 2024-03-25T19:19:49.307 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:19:49.307 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:49.307 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:19:49.307610+0000 2024-03-25T19:19:49.307 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:19:49.307610+0000 2024-03-25T19:19:49.307 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:49.307 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:49.307 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:19:49.307 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:19:49.307 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:19:49.307 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:49.308 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.307+0000 7f90896ba640 1 Processor -- start 2024-03-25T19:19:49.308 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.307+0000 7f90896ba640 1 -- start start 2024-03-25T19:19:49.308 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.307+0000 7f90896ba640 10 monclient: init 2024-03-25T19:19:49.308 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.308+0000 7f90896ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:49.308 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.308+0000 7f90896ba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:19:49.309 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.308+0000 7f90896ba640 10 monclient(hunting): picked mon.noname-b con 0x7f90840b4ff0 addr v1:172.21.15.6:6790/0 2024-03-25T19:19:49.309 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.308+0000 7f90896ba640 10 monclient(hunting): picked mon.noname-c con 0x7f90840b4670 addr v1:172.21.15.192:6789/0 2024-03-25T19:19:49.309 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.308+0000 7f90896ba640 10 monclient(hunting): picked mon.noname-a con 0x7f908413c480 addr v1:172.21.15.6:6789/0 2024-03-25T19:19:49.309 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.308+0000 7f90896ba640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90840ab0e0 con 0x7f908413c480 2024-03-25T19:19:49.309 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.308+0000 7f90896ba640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90840ab3c0 con 0x7f90840b4670 2024-03-25T19:19:49.309 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.308+0000 7f90896ba640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90840a9f30 con 0x7f90840b4ff0 2024-03-25T19:19:49.309 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.308+0000 7f90896ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:49.309 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.308+0000 7f90896ba640 10 monclient(hunting): authenticate will time out at 6293.548109s 2024-03-25T19:19:49.314 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.314+0000 7f90837fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f908413c480 0x7f908414dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38096/0 (socket says 172.21.15.6:38096) 2024-03-25T19:19:49.314 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.314+0000 7f90837fe640 1 -- 172.21.15.6:0/4060050809 learned_addr learned my addr 172.21.15.6:0/4060050809 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:19:49.336 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.336+0000 7f9081ffb640 1 -- 172.21.15.6:0/4060050809 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2855310292 0 0) 0x7f90840ab3c0 con 0x7f90840b4670 2024-03-25T19:19:49.336 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.336+0000 7f9081ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:19:49.337 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.336+0000 7f9081ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:49.337 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.336+0000 7f9081ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:49.337 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.336+0000 7f9081ffb640 10 monclient(hunting): my global_id is 4483 2024-03-25T19:19:49.337 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.336+0000 7f9081ffb640 1 -- 172.21.15.6:0/4060050809 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f906c003520 con 0x7f90840b4670 2024-03-25T19:19:49.346 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.346+0000 7f9081ffb640 1 -- 172.21.15.6:0/4060050809 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 844032578 0 0) 0x7f90840ab0e0 con 0x7f908413c480 2024-03-25T19:19:49.346 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.346+0000 7f9081ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:19:49.346 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.346+0000 7f9081ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:49.347 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.346+0000 7f9081ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:49.347 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.346+0000 7f9081ffb640 10 monclient(hunting): my global_id is 4479 2024-03-25T19:19:49.347 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.346+0000 7f9081ffb640 1 -- 172.21.15.6:0/4060050809 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90840ab3c0 con 0x7f908413c480 2024-03-25T19:19:49.355 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.354+0000 7f9081ffb640 1 -- 172.21.15.6:0/4060050809 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 727441162 0 0) 0x7f906c003520 con 0x7f90840b4670 2024-03-25T19:19:49.355 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.354+0000 7f9081ffb640 1 -- 172.21.15.6:0/4060050809 >> v1:172.21.15.6:6790/0 conn(0x7f90840b4ff0 legacy=0x7f908413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:49.355 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.354+0000 7f9081ffb640 1 -- 172.21.15.6:0/4060050809 >> v1:172.21.15.6:6789/0 conn(0x7f908413c480 legacy=0x7f908414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:49.355 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.354+0000 7f9081ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:19:49.355 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.354+0000 7f9081ffb640 1 monclient: found mon.noname-c 2024-03-25T19:19:49.355 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.354+0000 7f9081ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:19:49.355 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.354+0000 7f9081ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:19:49.355 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.354+0000 7f9081ffb640 1 -- 172.21.15.6:0/4060050809 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90840a9b70 con 0x7f90840b4670 2024-03-25T19:19:49.356 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.354+0000 7f9081ffb640 10 monclient: _finish_auth 0 2024-03-25T19:19:49.356 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.354+0000 7f9081ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:19:49.356 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.354+0000 7f9081ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:19:49.356 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.354+0000 7f90896ba640 5 monclient: authenticate success, global_id 4483 2024-03-25T19:19:49.356 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.354+0000 7f90896ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:19:49.356 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.354+0000 7f9081ffb640 1 -- 172.21.15.6:0/4060050809 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f9078001690 con 0x7f90840b4670 2024-03-25T19:19:49.356 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.354+0000 7f9081ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:49.357 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.354+0000 7f9081ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:19:49.357 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.354+0000 7f9081ffb640 10 monclient: dump: 2024-03-25T19:19:49.357 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:19:49.357 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:49.357 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:49.357 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:49.358 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:49.358 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:49.358 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:19:49.358 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:19:49.358 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:19:49.358 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:49.358 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.354+0000 7f90896ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:19:49.370 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.370+0000 7f9081ffb640 1 -- 172.21.15.6:0/4060050809 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90780018b0 con 0x7f90840b4670 2024-03-25T19:19:49.370 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.370+0000 7f9081ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:19:49.370 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.370+0000 7f9081ffb640 1 -- 172.21.15.6:0/4060050809 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f9078001c40 con 0x7f90840b4670 2024-03-25T19:19:49.370 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.370+0000 7f90896ba640 10 monclient: get_monmap_and_config success 2024-03-25T19:19:49.370 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.370+0000 7f90896ba640 4 set_mon_vals no callback set 2024-03-25T19:19:49.370 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.370+0000 7f9081ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:49.370 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.370+0000 7f9081ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:19:49.370 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.370+0000 7f9081ffb640 10 monclient: dump: 2024-03-25T19:19:49.370 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:19:49.370 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:49.371 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:49.371 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:49.371 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:49.371 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:49.371 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:19:49.371 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:19:49.371 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:19:49.371 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:49.373 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.370+0000 7f90896ba640 10 monclient: shutdown 2024-03-25T19:19:49.373 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.370+0000 7f90896ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:19:49.373 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.370+0000 7f90896ba640 1 -- 172.21.15.6:0/4060050809 >> v1:172.21.15.192:6789/0 conn(0x7f90840b4670 legacy=0x7f90841425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:49.373 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.370+0000 7f90896ba640 1 -- 172.21.15.6:0/4060050809 shutdown_connections 2024-03-25T19:19:49.373 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.370+0000 7f90896ba640 1 -- 172.21.15.6:0/4060050809 >> 172.21.15.6:0/4060050809 conn(0x7f90840617a0 msgr2=0x7f9084061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:19:49.373 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.371+0000 7f90896ba640 1 -- 172.21.15.6:0/4060050809 shutdown_connections 2024-03-25T19:19:49.373 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.371+0000 7f90896ba640 1 -- 172.21.15.6:0/4060050809 wait complete. 2024-03-25T19:19:49.373 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.371+0000 7f90896ba640 10 monclient: build_initial_monmap 2024-03-25T19:19:49.374 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.371+0000 7f90896ba640 10 monclient: monmap: 2024-03-25T19:19:49.374 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:19:49.374 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:19:49.374 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:19:49.374 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:19:49.374 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:49.374 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:19:49.374 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:19:49.374 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:19:49.374 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:19:49.374 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:49.374 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.372+0000 7f90896ba640 1 librados: starting msgr at 2024-03-25T19:19:49.374 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.372+0000 7f90896ba640 1 librados: starting objecter 2024-03-25T19:19:49.375 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.372+0000 7f90896ba640 1 Processor -- start 2024-03-25T19:19:49.375 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.372+0000 7f90896ba640 1 -- start start 2024-03-25T19:19:49.375 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.372+0000 7f90896ba640 1 librados: setting wanted keys 2024-03-25T19:19:49.375 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.372+0000 7f90896ba640 1 librados: calling monclient init 2024-03-25T19:19:49.375 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.372+0000 7f90896ba640 10 monclient: init 2024-03-25T19:19:49.375 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.372+0000 7f90896ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:49.375 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.372+0000 7f90896ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:19:49.375 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.372+0000 7f90896ba640 10 monclient(hunting): picked mon.noname-a con 0x7f90840b4670 addr v1:172.21.15.6:6789/0 2024-03-25T19:19:49.375 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.372+0000 7f90896ba640 10 monclient(hunting): picked mon.noname-c con 0x7f90840b4ff0 addr v1:172.21.15.6:6790/0 2024-03-25T19:19:49.375 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.372+0000 7f90896ba640 10 monclient(hunting): picked mon.noname-b con 0x7f908413c480 addr v1:172.21.15.192:6789/0 2024-03-25T19:19:49.375 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.372+0000 7f90896ba640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90840af4f0 con 0x7f90840b4670 2024-03-25T19:19:49.375 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.372+0000 7f90896ba640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90840ada40 con 0x7f908413c480 2024-03-25T19:19:49.375 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.372+0000 7f90896ba640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90840afcd0 con 0x7f90840b4ff0 2024-03-25T19:19:49.375 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.372+0000 7f90896ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:49.376 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.372+0000 7f90896ba640 10 monclient(hunting): authenticate will time out at 6293.612760s 2024-03-25T19:19:49.391 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.391+0000 7f9082ffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f90840b4670 0x7f908413a0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38156/0 (socket says 172.21.15.6:38156) 2024-03-25T19:19:49.391 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.391+0000 7f9082ffd640 1 -- 172.21.15.6:0/1326732003 learned_addr learned my addr 172.21.15.6:0/1326732003 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:19:49.393 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.393+0000 7f906b7fe640 10 client.?.objecter ms_handle_connect 0x7f90840b4670 2024-03-25T19:19:49.393 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.393+0000 7f906b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:49.395 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.395+0000 7f906b7fe640 10 client.?.objecter ms_handle_connect 0x7f908413c480 2024-03-25T19:19:49.395 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.395+0000 7f906b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:49.398 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.398+0000 7f906b7fe640 10 client.?.objecter ms_handle_connect 0x7f90840b4ff0 2024-03-25T19:19:49.398 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.398+0000 7f906b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:49.405 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.405+0000 7f906b7fe640 1 -- 172.21.15.6:0/1326732003 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3009560966 0 0) 0x7f90840af4f0 con 0x7f90840b4670 2024-03-25T19:19:49.405 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.405+0000 7f906b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:19:49.405 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.405+0000 7f906b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:49.405 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.405+0000 7f906b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:49.405 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.405+0000 7f906b7fe640 10 monclient(hunting): my global_id is 4482 2024-03-25T19:19:49.405 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.405+0000 7f906b7fe640 1 -- 172.21.15.6:0/1326732003 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9070004eb0 con 0x7f90840b4670 2024-03-25T19:19:49.414 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.413+0000 7f906b7fe640 1 -- 172.21.15.6:0/1326732003 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2856021632 0 0) 0x7f90840afcd0 con 0x7f90840b4ff0 2024-03-25T19:19:49.414 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.413+0000 7f906b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:19:49.414 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.413+0000 7f906b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:49.414 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.413+0000 7f906b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:49.414 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.413+0000 7f906b7fe640 10 monclient(hunting): my global_id is 4481 2024-03-25T19:19:49.414 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.413+0000 7f906b7fe640 1 -- 172.21.15.6:0/1326732003 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90840af4f0 con 0x7f90840b4ff0 2024-03-25T19:19:49.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.417+0000 7f906b7fe640 1 -- 172.21.15.6:0/1326732003 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 553637663 0 0) 0x7f90840ada40 con 0x7f908413c480 2024-03-25T19:19:49.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.417+0000 7f906b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:19:49.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.417+0000 7f906b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:49.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.417+0000 7f906b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:49.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.417+0000 7f906b7fe640 10 monclient(hunting): my global_id is 4486 2024-03-25T19:19:49.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.417+0000 7f906b7fe640 1 -- 172.21.15.6:0/1326732003 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90840afcd0 con 0x7f908413c480 2024-03-25T19:19:49.434 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.434+0000 7f906b7fe640 1 -- 172.21.15.6:0/1326732003 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 405972501 0 0) 0x7f90840afcd0 con 0x7f908413c480 2024-03-25T19:19:49.434 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.434+0000 7f906b7fe640 1 -- 172.21.15.6:0/1326732003 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90840ada40 con 0x7f908413c480 2024-03-25T19:19:49.435 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.434+0000 7f906b7fe640 1 -- 172.21.15.6:0/1326732003 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f907c001bd0 con 0x7f908413c480 2024-03-25T19:19:49.435 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.434+0000 7f906b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:49.435 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.434+0000 7f906b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:19:49.435 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.434+0000 7f906b7fe640 10 monclient(hunting): dump: 2024-03-25T19:19:49.435 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:19:49.435 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:49.435 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:49.435 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:49.435 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:49.435 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:49.435 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:19:49.435 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:19:49.435 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:19:49.435 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:49.436 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.434+0000 7f906b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:19:49.436 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.435+0000 7f906b7fe640 1 -- 172.21.15.6:0/1326732003 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1019124815 0 0) 0x7f9070004eb0 con 0x7f90840b4670 2024-03-25T19:19:49.436 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.435+0000 7f906b7fe640 1 -- 172.21.15.6:0/1326732003 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90840afcd0 con 0x7f90840b4670 2024-03-25T19:19:49.436 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.435+0000 7f906b7fe640 1 -- 172.21.15.6:0/1326732003 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f90740019d0 con 0x7f90840b4670 2024-03-25T19:19:49.436 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.435+0000 7f906b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:49.437 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.435+0000 7f906b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:19:49.437 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.435+0000 7f906b7fe640 10 monclient(hunting): dump: 2024-03-25T19:19:49.437 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:19:49.437 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:49.437 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:49.437 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:49.437 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:49.437 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:49.437 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:19:49.437 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:19:49.437 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:19:49.437 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:49.441 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.440+0000 7f906b7fe640 1 -- 172.21.15.6:0/1326732003 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1494096259 0 0) 0x7f90840af4f0 con 0x7f90840b4ff0 2024-03-25T19:19:49.441 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.440+0000 7f906b7fe640 1 -- 172.21.15.6:0/1326732003 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9070004eb0 con 0x7f90840b4ff0 2024-03-25T19:19:49.441 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.440+0000 7f906b7fe640 1 -- 172.21.15.6:0/1326732003 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f90780015d0 con 0x7f90840b4ff0 2024-03-25T19:19:49.441 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.441+0000 7f906b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:49.441 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.441+0000 7f906b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:19:49.441 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.441+0000 7f906b7fe640 10 monclient(hunting): dump: 2024-03-25T19:19:49.441 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:19:49.441 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:49.441 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:49.441 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:49.442 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:49.442 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:49.442 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:19:49.442 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:19:49.442 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:19:49.442 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:49.447 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.447+0000 7f906b7fe640 1 -- 172.21.15.6:0/1326732003 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2439238670 0 0) 0x7f90840ada40 con 0x7f908413c480 2024-03-25T19:19:49.447 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.447+0000 7f906b7fe640 1 -- 172.21.15.6:0/1326732003 >> v1:172.21.15.6:6790/0 conn(0x7f90840b4ff0 legacy=0x7f90840b0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:49.447 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.447+0000 7f906b7fe640 1 -- 172.21.15.6:0/1326732003 >> v1:172.21.15.6:6789/0 conn(0x7f90840b4670 legacy=0x7f908413a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:49.447 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.447+0000 7f906b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:19:49.448 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.447+0000 7f906b7fe640 1 monclient: found mon.b 2024-03-25T19:19:49.448 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.447+0000 7f906b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:19:49.448 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.447+0000 7f906b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:19:49.448 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.447+0000 7f906b7fe640 1 -- 172.21.15.6:0/1326732003 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90840affb0 con 0x7f908413c480 2024-03-25T19:19:49.448 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.447+0000 7f906b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:19:49.448 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.447+0000 7f906b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:19:49.448 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.447+0000 7f906b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:19:49.448 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.447+0000 7f90896ba640 5 monclient: authenticate success, global_id 4486 2024-03-25T19:19:49.448 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.447+0000 7f90896ba640 10 monclient: _renew_subs 2024-03-25T19:19:49.448 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.447+0000 7f90896ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:19:49.448 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.447+0000 7f90896ba640 1 -- 172.21.15.6:0/1326732003 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90840b0290 con 0x7f908413c480 2024-03-25T19:19:49.448 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.447+0000 7f90896ba640 10 client.4486.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:19:49.448 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.447+0000 7f90896ba640 10 monclient: _renew_subs 2024-03-25T19:19:49.448 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.447+0000 7f90896ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:19:49.448 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.447+0000 7f90896ba640 1 -- 172.21.15.6:0/1326732003 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f90840aefd0 con 0x7f908413c480 2024-03-25T19:19:49.449 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.447+0000 7f90896ba640 1 librados: init done 2024-03-25T19:19:49.449 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.448+0000 7f90896ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:49.449 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.448+0000 7f90896ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:49.449 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.448+0000 7f90896ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:19:49.449 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.448+0000 7f90896ba640 1 -- 172.21.15.6:0/1326732003 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f908406dbd0 con 0x7f908413c480 2024-03-25T19:19:49.732 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.732+0000 7f906b7fe640 1 -- 172.21.15.6:0/1326732003 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f907c001fe0 con 0x7f908413c480 2024-03-25T19:19:49.732 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.732+0000 7f906b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:19:49.732 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.732+0000 7f906b7fe640 1 -- 172.21.15.6:0/1326732003 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f907c004d00 con 0x7f908413c480 2024-03-25T19:19:49.733 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.732+0000 7f906b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:49.733 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.732+0000 7f906b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:19:49.733 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.732+0000 7f906b7fe640 10 monclient: dump: 2024-03-25T19:19:49.733 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:19:49.733 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:49.733 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:49.733 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:49.733 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:49.733 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:49.733 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:19:49.733 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:19:49.733 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:19:49.733 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:49.733 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.732+0000 7f90817fa640 4 set_mon_vals no callback set 2024-03-25T19:19:49.734 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.733+0000 7f906b7fe640 1 -- 172.21.15.6:0/1326732003 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f907c01acd0 con 0x7f908413c480 2024-03-25T19:19:49.734 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.734+0000 7f9082ffd640 10 monclient: get_auth_request con 0x7f907006e0b0 auth_method 0 2024-03-25T19:19:49.734 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.734+0000 7f906b7fe640 10 client.4486.objecter ms_handle_connect 0x7f907006e0b0 2024-03-25T19:19:49.738 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.738+0000 7f906b7fe640 1 -- 172.21.15.6:0/1326732003 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f907c054a20 con 0x7f908413c480 2024-03-25T19:19:49.738 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.738+0000 7f906b7fe640 10 client.4486.objecter ms_dispatch 0x7f9084000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:19:49.738 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.738+0000 7f906b7fe640 3 client.4486.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:19:49.738 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.738+0000 7f906b7fe640 3 client.4486.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:19:49.738 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.738+0000 7f906b7fe640 20 client.4486.objecter dump_active .. 0 homeless 2024-03-25T19:19:49.738 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.738+0000 7f906b7fe640 1 -- 172.21.15.6:0/1326732003 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f907c050360 con 0x7f908413c480 2024-03-25T19:19:49.738 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.738+0000 7f906b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:49.738 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.738+0000 7f906b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:19:49.983 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.983+0000 7f90896ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:19:49.983 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.983+0000 7f90896ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:19:49.983 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.983+0000 7f90896ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:19:49.983 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:49.983+0000 7f90896ba640 1 -- 172.21.15.6:0/1326732003 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90840688c0 con 0x7f908413c480 2024-03-25T19:19:50.006 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:50.006+0000 7f906b7fe640 1 -- 172.21.15.6:0/1326732003 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f907c052460 con 0x7f908413c480 2024-03-25T19:19:50.006 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:50.006+0000 7f906b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:19:50.006 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:50.006+0000 7f906b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:19:50.007 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:19:50.007 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:19:50.010 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:50.009+0000 7f90896ba640 10 librados: watch_flush enter 2024-03-25T19:19:50.010 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:50.009+0000 7f90896ba640 10 librados: watch_flush exit 2024-03-25T19:19:50.010 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:50.009+0000 7f90896ba640 20 client.4486.objecter shutdown clearing up homeless session... 2024-03-25T19:19:50.010 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:50.009+0000 7f90896ba640 10 client.4486.objecter successfully canceled tick 2024-03-25T19:19:50.010 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:50.009+0000 7f90896ba640 1 -- 172.21.15.6:0/1326732003 >> v1:172.21.15.6:6800/2936652650 conn(0x7f907006e0b0 legacy=0x7f9070070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:50.010 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:50.009+0000 7f90896ba640 10 monclient: shutdown 2024-03-25T19:19:50.011 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:50.009+0000 7f90896ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:19:50.011 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:50.009+0000 7f90896ba640 1 -- 172.21.15.6:0/1326732003 >> v1:172.21.15.192:6789/0 conn(0x7f908413c480 legacy=0x7f90840b10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:50.011 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:50.009+0000 7f90896ba640 1 -- 172.21.15.6:0/1326732003 shutdown_connections 2024-03-25T19:19:50.011 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:50.009+0000 7f90896ba640 1 -- 172.21.15.6:0/1326732003 >> 172.21.15.6:0/1326732003 conn(0x7f90840617a0 msgr2=0x7f90840b71c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:19:50.011 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:50.010+0000 7f90896ba640 1 -- 172.21.15.6:0/1326732003 shutdown_connections 2024-03-25T19:19:50.011 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:50.010+0000 7f90896ba640 1 -- 172.21.15.6:0/1326732003 wait complete. 2024-03-25T19:19:50.011 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:50.010+0000 7f90896ba640 1 librados: shutdown 2024-03-25T19:19:50.020 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:19:56.826 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:19:56.940 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.940+0000 7f556d689640 10 monclient: get_monmap_and_config 2024-03-25T19:19:56.941 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.940+0000 7f556d689640 10 monclient: build_initial_monmap 2024-03-25T19:19:56.941 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.940+0000 7f556d689640 10 monclient: monmap: 2024-03-25T19:19:56.941 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:19:56.941 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:56.941 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:19:56.941753+0000 2024-03-25T19:19:56.941 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:19:56.941753+0000 2024-03-25T19:19:56.941 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:56.941 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:56.941 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:19:56.941 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:19:56.941 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:19:56.942 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:56.942 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.941+0000 7f556d689640 1 Processor -- start 2024-03-25T19:19:56.943 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.941+0000 7f556d689640 1 -- start start 2024-03-25T19:19:56.943 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.941+0000 7f556d689640 10 monclient: init 2024-03-25T19:19:56.943 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.942+0000 7f556d689640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:56.943 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.942+0000 7f556d689640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:19:56.943 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.942+0000 7f556d689640 10 monclient(hunting): picked mon.noname-c con 0x7f55680b4ff0 addr v1:172.21.15.192:6789/0 2024-03-25T19:19:56.943 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.942+0000 7f556d689640 10 monclient(hunting): picked mon.noname-b con 0x7f55680b4670 addr v1:172.21.15.6:6790/0 2024-03-25T19:19:56.943 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.942+0000 7f556d689640 10 monclient(hunting): picked mon.noname-a con 0x7f556813c480 addr v1:172.21.15.6:6789/0 2024-03-25T19:19:56.943 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.942+0000 7f556d689640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55680ab0e0 con 0x7f556813c480 2024-03-25T19:19:56.943 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.942+0000 7f556d689640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55680ab3c0 con 0x7f55680b4ff0 2024-03-25T19:19:56.943 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.942+0000 7f556d689640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55680a9f30 con 0x7f55680b4670 2024-03-25T19:19:56.943 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.942+0000 7f556d689640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:56.943 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.942+0000 7f556d689640 10 monclient(hunting): authenticate will time out at 6301.182348s 2024-03-25T19:19:56.951 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.951+0000 7f5566ffd640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f55680b4ff0 0x7f556813bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:50504/0 (socket says 172.21.15.6:50504) 2024-03-25T19:19:56.951 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.951+0000 7f5566ffd640 1 -- 172.21.15.6:0/3009593481 learned_addr learned my addr 172.21.15.6:0/3009593481 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:19:56.969 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.968+0000 7f5565ffb640 1 -- 172.21.15.6:0/3009593481 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 778870619 0 0) 0x7f55680ab3c0 con 0x7f55680b4ff0 2024-03-25T19:19:56.969 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.969+0000 7f5565ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:19:56.969 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.969+0000 7f5565ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:56.969 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.969+0000 7f5565ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:56.969 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.969+0000 7f5565ffb640 10 monclient(hunting): my global_id is 4489 2024-03-25T19:19:56.969 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.969+0000 7f5565ffb640 1 -- 172.21.15.6:0/3009593481 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5550003520 con 0x7f55680b4ff0 2024-03-25T19:19:56.978 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.977+0000 7f5565ffb640 1 -- 172.21.15.6:0/3009593481 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 339701908 0 0) 0x7f5550003520 con 0x7f55680b4ff0 2024-03-25T19:19:56.978 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.977+0000 7f5565ffb640 1 -- 172.21.15.6:0/3009593481 >> v1:172.21.15.6:6790/0 conn(0x7f55680b4670 legacy=0x7f55681425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:56.978 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.977+0000 7f5565ffb640 1 -- 172.21.15.6:0/3009593481 >> v1:172.21.15.6:6789/0 conn(0x7f556813c480 legacy=0x7f556814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:56.978 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.977+0000 7f5565ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:19:56.978 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.977+0000 7f5565ffb640 1 monclient: found mon.noname-c 2024-03-25T19:19:56.978 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.977+0000 7f5565ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:19:56.978 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.977+0000 7f5565ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:19:56.978 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.977+0000 7f5565ffb640 1 -- 172.21.15.6:0/3009593481 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55680a9b70 con 0x7f55680b4ff0 2024-03-25T19:19:56.978 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.977+0000 7f5565ffb640 10 monclient: _finish_auth 0 2024-03-25T19:19:56.978 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.977+0000 7f5565ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:19:56.979 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.977+0000 7f5565ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:19:56.979 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.977+0000 7f5565ffb640 1 -- 172.21.15.6:0/3009593481 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f5558001690 con 0x7f55680b4ff0 2024-03-25T19:19:56.979 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.977+0000 7f5565ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:56.979 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.978+0000 7f5565ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:19:56.979 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.978+0000 7f5565ffb640 10 monclient: dump: 2024-03-25T19:19:56.979 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:19:56.979 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:56.979 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:56.979 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:56.979 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:56.979 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:56.979 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:19:56.979 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:19:56.979 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:19:56.979 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:56.980 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.978+0000 7f556d689640 5 monclient: authenticate success, global_id 4489 2024-03-25T19:19:56.980 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:56.978+0000 7f556d689640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:19:57.008 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.008+0000 7f5565ffb640 1 -- 172.21.15.6:0/3009593481 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f55580017f0 con 0x7f55680b4ff0 2024-03-25T19:19:57.008 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.008+0000 7f5565ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:19:57.008 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.008+0000 7f5565ffb640 1 -- 172.21.15.6:0/3009593481 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f5558001c00 con 0x7f55680b4ff0 2024-03-25T19:19:57.009 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.008+0000 7f5565ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:57.009 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.008+0000 7f5565ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:19:57.009 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.008+0000 7f5565ffb640 10 monclient: dump: 2024-03-25T19:19:57.009 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:19:57.009 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:57.009 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:57.009 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:57.009 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:57.009 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:57.009 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:19:57.009 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:19:57.009 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:19:57.009 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:57.009 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.008+0000 7f556d689640 10 monclient: get_monmap_and_config success 2024-03-25T19:19:57.010 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.008+0000 7f556d689640 4 set_mon_vals no callback set 2024-03-25T19:19:57.011 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.009+0000 7f556d689640 10 monclient: shutdown 2024-03-25T19:19:57.011 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.009+0000 7f556d689640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:19:57.011 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.009+0000 7f556d689640 1 -- 172.21.15.6:0/3009593481 >> v1:172.21.15.192:6789/0 conn(0x7f55680b4ff0 legacy=0x7f556813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:57.011 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.009+0000 7f556d689640 1 -- 172.21.15.6:0/3009593481 shutdown_connections 2024-03-25T19:19:57.011 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.009+0000 7f556d689640 1 -- 172.21.15.6:0/3009593481 >> 172.21.15.6:0/3009593481 conn(0x7f55680617a0 msgr2=0x7f5568061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:19:57.011 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.009+0000 7f556d689640 1 -- 172.21.15.6:0/3009593481 shutdown_connections 2024-03-25T19:19:57.011 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.009+0000 7f556d689640 1 -- 172.21.15.6:0/3009593481 wait complete. 2024-03-25T19:19:57.011 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.010+0000 7f556d689640 10 monclient: build_initial_monmap 2024-03-25T19:19:57.011 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.010+0000 7f556d689640 10 monclient: monmap: 2024-03-25T19:19:57.011 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:19:57.012 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:19:57.012 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:19:57.012 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:19:57.012 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:57.012 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:19:57.012 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:19:57.012 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:19:57.012 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:19:57.012 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:57.012 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.010+0000 7f556d689640 1 librados: starting msgr at 2024-03-25T19:19:57.012 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.010+0000 7f556d689640 1 librados: starting objecter 2024-03-25T19:19:57.012 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.011+0000 7f556d689640 1 Processor -- start 2024-03-25T19:19:57.012 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.011+0000 7f556d689640 1 -- start start 2024-03-25T19:19:57.012 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.011+0000 7f556d689640 1 librados: setting wanted keys 2024-03-25T19:19:57.013 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.011+0000 7f556d689640 1 librados: calling monclient init 2024-03-25T19:19:57.013 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.011+0000 7f556d689640 10 monclient: init 2024-03-25T19:19:57.013 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.011+0000 7f556d689640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:57.013 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.011+0000 7f556d689640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:19:57.013 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.011+0000 7f556d689640 10 monclient(hunting): picked mon.noname-a con 0x7f55680b4670 addr v1:172.21.15.6:6789/0 2024-03-25T19:19:57.014 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.011+0000 7f556d689640 10 monclient(hunting): picked mon.noname-b con 0x7f55680b4ff0 addr v1:172.21.15.192:6789/0 2024-03-25T19:19:57.014 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.011+0000 7f556d689640 10 monclient(hunting): picked mon.noname-c con 0x7f556813c480 addr v1:172.21.15.6:6790/0 2024-03-25T19:19:57.014 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.011+0000 7f556d689640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5568143c80 con 0x7f55680b4670 2024-03-25T19:19:57.014 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.011+0000 7f556d689640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5568146570 con 0x7f55680b4ff0 2024-03-25T19:19:57.014 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.012+0000 7f556d689640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5568130cc0 con 0x7f556813c480 2024-03-25T19:19:57.014 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.012+0000 7f556d689640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:57.014 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.012+0000 7f556d689640 10 monclient(hunting): authenticate will time out at 6301.251785s 2024-03-25T19:19:57.014 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.014+0000 7f55667fc640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f55680b4ff0 0x7f5568145e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:50552/0 (socket says 172.21.15.6:50552) 2024-03-25T19:19:57.014 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.014+0000 7f55667fc640 1 -- 172.21.15.6:0/3748656456 learned_addr learned my addr 172.21.15.6:0/3748656456 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:19:57.017 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.017+0000 7f554f7fe640 10 client.?.objecter ms_handle_connect 0x7f55680b4ff0 2024-03-25T19:19:57.017 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.017+0000 7f554f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:57.029 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.029+0000 7f554f7fe640 1 -- 172.21.15.6:0/3748656456 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1737941255 0 0) 0x7f5568146570 con 0x7f55680b4ff0 2024-03-25T19:19:57.029 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.029+0000 7f554f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:19:57.029 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.029+0000 7f554f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:57.029 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.029+0000 7f554f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:57.029 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.029+0000 7f554f7fe640 10 monclient(hunting): my global_id is 4492 2024-03-25T19:19:57.029 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.029+0000 7f554f7fe640 1 -- 172.21.15.6:0/3748656456 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5554004ee0 con 0x7f55680b4ff0 2024-03-25T19:19:57.029 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.029+0000 7f554f7fe640 10 client.?.objecter ms_handle_connect 0x7f556813c480 2024-03-25T19:19:57.029 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.029+0000 7f554f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:57.035 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.036+0000 7f554f7fe640 1 -- 172.21.15.6:0/3748656456 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3964014925 0 0) 0x7f5568130cc0 con 0x7f556813c480 2024-03-25T19:19:57.036 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.036+0000 7f554f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:19:57.036 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.036+0000 7f554f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:57.036 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.036+0000 7f554f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:57.036 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.036+0000 7f554f7fe640 10 monclient(hunting): my global_id is 4487 2024-03-25T19:19:57.036 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.036+0000 7f554f7fe640 1 -- 172.21.15.6:0/3748656456 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5568146570 con 0x7f556813c480 2024-03-25T19:19:57.042 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.041+0000 7f554f7fe640 1 -- 172.21.15.6:0/3748656456 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 680065945 0 0) 0x7f5554004ee0 con 0x7f55680b4ff0 2024-03-25T19:19:57.042 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.041+0000 7f554f7fe640 1 -- 172.21.15.6:0/3748656456 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5568130cc0 con 0x7f55680b4ff0 2024-03-25T19:19:57.042 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.041+0000 7f554f7fe640 1 -- 172.21.15.6:0/3748656456 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f555c001a30 con 0x7f55680b4ff0 2024-03-25T19:19:57.042 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.041+0000 7f554f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:57.042 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.041+0000 7f554f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:19:57.042 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.041+0000 7f554f7fe640 10 monclient(hunting): dump: 2024-03-25T19:19:57.043 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:19:57.043 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:57.043 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:57.043 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:57.043 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:57.043 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:57.043 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:19:57.043 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:19:57.043 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:19:57.043 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:57.043 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.042+0000 7f554f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:19:57.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.045+0000 7f554f7fe640 10 client.?.objecter ms_handle_connect 0x7f55680b4670 2024-03-25T19:19:57.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.045+0000 7f554f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:57.052 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.052+0000 7f554f7fe640 1 -- 172.21.15.6:0/3748656456 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3565559943 0 0) 0x7f5568146570 con 0x7f556813c480 2024-03-25T19:19:57.052 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.052+0000 7f554f7fe640 1 -- 172.21.15.6:0/3748656456 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5554004ee0 con 0x7f556813c480 2024-03-25T19:19:57.052 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.052+0000 7f554f7fe640 1 -- 172.21.15.6:0/3748656456 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f5560001b00 con 0x7f556813c480 2024-03-25T19:19:57.052 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.052+0000 7f554f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:57.052 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.052+0000 7f554f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:19:57.052 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.052+0000 7f554f7fe640 10 monclient(hunting): dump: 2024-03-25T19:19:57.052 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:19:57.052 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:57.052 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:57.053 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:57.053 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:57.053 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:57.053 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:19:57.053 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:19:57.053 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:19:57.053 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:57.058 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.058+0000 7f554f7fe640 1 -- 172.21.15.6:0/3748656456 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 988096466 0 0) 0x7f5568130cc0 con 0x7f55680b4ff0 2024-03-25T19:19:57.058 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.058+0000 7f554f7fe640 1 -- 172.21.15.6:0/3748656456 >> v1:172.21.15.6:6790/0 conn(0x7f556813c480 legacy=0x7f55681e2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:57.058 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.058+0000 7f554f7fe640 1 -- 172.21.15.6:0/3748656456 >> v1:172.21.15.6:6789/0 conn(0x7f55680b4670 legacy=0x7f5568145750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:57.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.058+0000 7f554f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:19:57.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.058+0000 7f554f7fe640 1 monclient: found mon.b 2024-03-25T19:19:57.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.058+0000 7f554f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:19:57.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.058+0000 7f554f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:19:57.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.058+0000 7f554f7fe640 1 -- 172.21.15.6:0/3748656456 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5568144a00 con 0x7f55680b4ff0 2024-03-25T19:19:57.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.058+0000 7f554f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:19:57.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.058+0000 7f554f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:19:57.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.058+0000 7f554f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:19:57.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.058+0000 7f556d689640 5 monclient: authenticate success, global_id 4492 2024-03-25T19:19:57.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.058+0000 7f556d689640 10 monclient: _renew_subs 2024-03-25T19:19:57.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.058+0000 7f556d689640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:19:57.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.058+0000 7f556d689640 1 -- 172.21.15.6:0/3748656456 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5568144ce0 con 0x7f55680b4ff0 2024-03-25T19:19:57.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.058+0000 7f556d689640 10 client.4492.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:19:57.060 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.058+0000 7f556d689640 10 monclient: _renew_subs 2024-03-25T19:19:57.060 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.058+0000 7f556d689640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:19:57.060 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.058+0000 7f556d689640 1 -- 172.21.15.6:0/3748656456 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f556813eec0 con 0x7f55680b4ff0 2024-03-25T19:19:57.060 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.058+0000 7f556d689640 1 librados: init done 2024-03-25T19:19:57.060 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.059+0000 7f556d689640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:57.060 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.059+0000 7f556d689640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:57.060 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.059+0000 7f556d689640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:19:57.061 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.059+0000 7f556d689640 1 -- 172.21.15.6:0/3748656456 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f556806dbd0 con 0x7f55680b4ff0 2024-03-25T19:19:57.336 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.336+0000 7f554f7fe640 1 -- 172.21.15.6:0/3748656456 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f555c001c70 con 0x7f55680b4ff0 2024-03-25T19:19:57.336 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.336+0000 7f554f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:19:57.336 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.336+0000 7f554f7fe640 1 -- 172.21.15.6:0/3748656456 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f555c003a60 con 0x7f55680b4ff0 2024-03-25T19:19:57.336 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.336+0000 7f554f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:57.336 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.336+0000 7f554f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:19:57.336 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.336+0000 7f55657fa640 4 set_mon_vals no callback set 2024-03-25T19:19:57.337 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.336+0000 7f554f7fe640 10 monclient: dump: 2024-03-25T19:19:57.337 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:19:57.337 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:19:57.337 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:57.337 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:19:57.337 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:57.337 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:19:57.337 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:19:57.337 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:19:57.337 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:19:57.337 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:19:57.338 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.337+0000 7f554f7fe640 1 -- 172.21.15.6:0/3748656456 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f555c003c40 con 0x7f55680b4ff0 2024-03-25T19:19:57.338 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.337+0000 7f5566ffd640 10 monclient: get_auth_request con 0x7f555406de70 auth_method 0 2024-03-25T19:19:57.338 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.337+0000 7f554f7fe640 10 client.4492.objecter ms_handle_connect 0x7f555406de70 2024-03-25T19:19:57.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.342+0000 7f554f7fe640 1 -- 172.21.15.6:0/3748656456 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f555c054850 con 0x7f55680b4ff0 2024-03-25T19:19:57.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.342+0000 7f554f7fe640 10 client.4492.objecter ms_dispatch 0x7f5568000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:19:57.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.342+0000 7f554f7fe640 3 client.4492.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:19:57.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.342+0000 7f554f7fe640 3 client.4492.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:19:57.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.342+0000 7f554f7fe640 20 client.4492.objecter dump_active .. 0 homeless 2024-03-25T19:19:57.343 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.343+0000 7f554f7fe640 1 -- 172.21.15.6:0/3748656456 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f555c050360 con 0x7f55680b4ff0 2024-03-25T19:19:57.343 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.343+0000 7f554f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:57.343 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.343+0000 7f554f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:19:57.573 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.573+0000 7f556d689640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:19:57.573 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.573+0000 7f556d689640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:19:57.573 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.573+0000 7f556d689640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:19:57.573 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.573+0000 7f556d689640 1 -- 172.21.15.6:0/3748656456 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f556813d900 con 0x7f55680b4ff0 2024-03-25T19:19:57.609 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.609+0000 7f554f7fe640 1 -- 172.21.15.6:0/3748656456 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f555c052290 con 0x7f55680b4ff0 2024-03-25T19:19:57.609 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.609+0000 7f554f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:19:57.609 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.609+0000 7f554f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:19:57.609 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:19:57.609 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:19:57.612 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.611+0000 7f556d689640 10 librados: watch_flush enter 2024-03-25T19:19:57.612 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.611+0000 7f556d689640 10 librados: watch_flush exit 2024-03-25T19:19:57.612 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.611+0000 7f556d689640 20 client.4492.objecter shutdown clearing up homeless session... 2024-03-25T19:19:57.612 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.611+0000 7f556d689640 10 client.4492.objecter successfully canceled tick 2024-03-25T19:19:57.612 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.611+0000 7f556d689640 1 -- 172.21.15.6:0/3748656456 >> v1:172.21.15.6:6800/2936652650 conn(0x7f555406de70 legacy=0x7f5554070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:57.612 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.611+0000 7f556d689640 10 monclient: shutdown 2024-03-25T19:19:57.612 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.611+0000 7f556d689640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:19:57.613 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.611+0000 7f556d689640 1 -- 172.21.15.6:0/3748656456 >> v1:172.21.15.192:6789/0 conn(0x7f55680b4ff0 legacy=0x7f5568145e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:57.613 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.611+0000 7f556d689640 1 -- 172.21.15.6:0/3748656456 shutdown_connections 2024-03-25T19:19:57.613 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.611+0000 7f556d689640 1 -- 172.21.15.6:0/3748656456 >> 172.21.15.6:0/3748656456 conn(0x7f55680617a0 msgr2=0x7f55680b2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:19:57.613 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.612+0000 7f556d689640 1 -- 172.21.15.6:0/3748656456 shutdown_connections 2024-03-25T19:19:57.613 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.612+0000 7f556d689640 1 -- 172.21.15.6:0/3748656456 wait complete. 2024-03-25T19:19:57.613 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:19:57.612+0000 7f556d689640 1 librados: shutdown 2024-03-25T19:19:57.624 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:20:04.431 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:20:04.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.545+0000 7f2be8c9a640 10 monclient: get_monmap_and_config 2024-03-25T19:20:04.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.545+0000 7f2be8c9a640 10 monclient: build_initial_monmap 2024-03-25T19:20:04.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.545+0000 7f2be8c9a640 10 monclient: monmap: 2024-03-25T19:20:04.545 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:20:04.545 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:04.545 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:20:04.546159+0000 2024-03-25T19:20:04.545 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:20:04.546159+0000 2024-03-25T19:20:04.546 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:04.546 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:04.546 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:20:04.546 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:20:04.546 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:20:04.546 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:04.547 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.545+0000 7f2be8c9a640 1 Processor -- start 2024-03-25T19:20:04.547 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.546+0000 7f2be8c9a640 1 -- start start 2024-03-25T19:20:04.547 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.546+0000 7f2be8c9a640 10 monclient: init 2024-03-25T19:20:04.547 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.546+0000 7f2be8c9a640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:04.547 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.546+0000 7f2be8c9a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:20:04.547 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.546+0000 7f2be8c9a640 10 monclient(hunting): picked mon.noname-c con 0x7f2be40b4ff0 addr v1:172.21.15.192:6789/0 2024-03-25T19:20:04.548 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.546+0000 7f2be8c9a640 10 monclient(hunting): picked mon.noname-b con 0x7f2be40b4670 addr v1:172.21.15.6:6790/0 2024-03-25T19:20:04.548 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.546+0000 7f2be8c9a640 10 monclient(hunting): picked mon.noname-a con 0x7f2be413c480 addr v1:172.21.15.6:6789/0 2024-03-25T19:20:04.548 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.546+0000 7f2be8c9a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2be40ab0e0 con 0x7f2be413c480 2024-03-25T19:20:04.548 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.546+0000 7f2be8c9a640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2be40ab3c0 con 0x7f2be40b4ff0 2024-03-25T19:20:04.548 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.546+0000 7f2be8c9a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2be40a9f30 con 0x7f2be40b4670 2024-03-25T19:20:04.548 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.546+0000 7f2be8c9a640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:04.548 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.546+0000 7f2be8c9a640 10 monclient(hunting): authenticate will time out at 6308.786681s 2024-03-25T19:20:04.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.552+0000 7f2be2d9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2be413c480 0x7f2be414dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39762/0 (socket says 172.21.15.6:39762) 2024-03-25T19:20:04.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.552+0000 7f2be2d9d640 1 -- 172.21.15.6:0/2707248798 learned_addr learned my addr 172.21.15.6:0/2707248798 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:20:04.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.559+0000 7f2be159a640 1 -- 172.21.15.6:0/2707248798 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 171333736 0 0) 0x7f2be40ab0e0 con 0x7f2be413c480 2024-03-25T19:20:04.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.559+0000 7f2be159a640 10 monclient(hunting): _negotiate 2024-03-25T19:20:04.560 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.559+0000 7f2be159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:04.560 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.559+0000 7f2be159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:04.560 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.559+0000 7f2be159a640 10 monclient(hunting): my global_id is 4491 2024-03-25T19:20:04.560 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.559+0000 7f2be159a640 1 -- 172.21.15.6:0/2707248798 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bcc003520 con 0x7f2be413c480 2024-03-25T19:20:04.560 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.560+0000 7f2be159a640 1 -- 172.21.15.6:0/2707248798 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1352110565 0 0) 0x7f2be40ab3c0 con 0x7f2be40b4ff0 2024-03-25T19:20:04.560 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.560+0000 7f2be159a640 10 monclient(hunting): _negotiate 2024-03-25T19:20:04.560 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.560+0000 7f2be159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:04.561 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.560+0000 7f2be159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:04.561 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.560+0000 7f2be159a640 10 monclient(hunting): my global_id is 4495 2024-03-25T19:20:04.561 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.560+0000 7f2be159a640 1 -- 172.21.15.6:0/2707248798 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2be40ab0e0 con 0x7f2be40b4ff0 2024-03-25T19:20:04.570 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.569+0000 7f2be159a640 1 -- 172.21.15.6:0/2707248798 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 571954669 0 0) 0x7f2bcc003520 con 0x7f2be413c480 2024-03-25T19:20:04.570 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.569+0000 7f2be159a640 1 -- 172.21.15.6:0/2707248798 >> v1:172.21.15.6:6790/0 conn(0x7f2be40b4670 legacy=0x7f2be41425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:04.570 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.569+0000 7f2be159a640 1 -- 172.21.15.6:0/2707248798 >> v1:172.21.15.192:6789/0 conn(0x7f2be40b4ff0 legacy=0x7f2be413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:04.570 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.570+0000 7f2be159a640 10 monclient: _finish_hunting 0 2024-03-25T19:20:04.570 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.570+0000 7f2be159a640 1 monclient: found mon.noname-a 2024-03-25T19:20:04.570 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.570+0000 7f2be159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:04.570 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.570+0000 7f2be159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-03-25T19:20:04.570 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.570+0000 7f2be159a640 1 -- 172.21.15.6:0/2707248798 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be40a9b70 con 0x7f2be413c480 2024-03-25T19:20:04.570 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.570+0000 7f2be159a640 10 monclient: _finish_auth 0 2024-03-25T19:20:04.570 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.570+0000 7f2be159a640 10 monclient: _check_auth_tickets 2024-03-25T19:20:04.571 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.570+0000 7f2be159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:04.571 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.570+0000 7f2be159a640 1 -- 172.21.15.6:0/2707248798 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f2bdc001a10 con 0x7f2be413c480 2024-03-25T19:20:04.571 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.570+0000 7f2be159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:04.571 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.570+0000 7f2be159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:20:04.571 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.570+0000 7f2be159a640 10 monclient: dump: 2024-03-25T19:20:04.571 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:20:04.571 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:04.571 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:04.571 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:04.571 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:04.571 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:04.571 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:20:04.571 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:20:04.571 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:20:04.571 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:04.572 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.570+0000 7f2be8c9a640 5 monclient: authenticate success, global_id 4491 2024-03-25T19:20:04.572 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.570+0000 7f2be8c9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:04.575 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.575+0000 7f2be159a640 1 -- 172.21.15.6:0/2707248798 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2bdc001b70 con 0x7f2be413c480 2024-03-25T19:20:04.575 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.575+0000 7f2be159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:04.576 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.575+0000 7f2be159a640 1 -- 172.21.15.6:0/2707248798 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f2bdc001f80 con 0x7f2be413c480 2024-03-25T19:20:04.576 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.575+0000 7f2be159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:04.576 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.575+0000 7f2be159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:20:04.576 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.575+0000 7f2be159a640 10 monclient: dump: 2024-03-25T19:20:04.576 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:20:04.576 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:04.576 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:04.576 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:04.576 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:04.576 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:04.576 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:20:04.576 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:20:04.576 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:20:04.577 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:04.577 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.575+0000 7f2be8c9a640 10 monclient: get_monmap_and_config success 2024-03-25T19:20:04.577 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.575+0000 7f2be8c9a640 4 set_mon_vals no callback set 2024-03-25T19:20:04.578 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.576+0000 7f2be8c9a640 10 monclient: shutdown 2024-03-25T19:20:04.578 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.576+0000 7f2be8c9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:04.578 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.576+0000 7f2be8c9a640 1 -- 172.21.15.6:0/2707248798 >> v1:172.21.15.6:6789/0 conn(0x7f2be413c480 legacy=0x7f2be414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:04.578 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.576+0000 7f2be8c9a640 1 -- 172.21.15.6:0/2707248798 shutdown_connections 2024-03-25T19:20:04.578 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.576+0000 7f2be8c9a640 1 -- 172.21.15.6:0/2707248798 >> 172.21.15.6:0/2707248798 conn(0x7f2be40617a0 msgr2=0x7f2be4061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:04.578 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.576+0000 7f2be8c9a640 1 -- 172.21.15.6:0/2707248798 shutdown_connections 2024-03-25T19:20:04.578 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.576+0000 7f2be8c9a640 1 -- 172.21.15.6:0/2707248798 wait complete. 2024-03-25T19:20:04.579 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.577+0000 7f2be8c9a640 10 monclient: build_initial_monmap 2024-03-25T19:20:04.579 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.577+0000 7f2be8c9a640 10 monclient: monmap: 2024-03-25T19:20:04.579 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:20:04.579 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:20:04.579 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:20:04.579 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:20:04.579 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:04.579 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:20:04.579 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:20:04.579 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:20:04.580 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:20:04.580 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:04.580 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.578+0000 7f2be8c9a640 1 librados: starting msgr at 2024-03-25T19:20:04.580 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.578+0000 7f2be8c9a640 1 librados: starting objecter 2024-03-25T19:20:04.580 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.578+0000 7f2be8c9a640 1 Processor -- start 2024-03-25T19:20:04.580 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.578+0000 7f2be8c9a640 1 -- start start 2024-03-25T19:20:04.580 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.578+0000 7f2be8c9a640 1 librados: setting wanted keys 2024-03-25T19:20:04.580 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.578+0000 7f2be8c9a640 1 librados: calling monclient init 2024-03-25T19:20:04.580 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.578+0000 7f2be8c9a640 10 monclient: init 2024-03-25T19:20:04.581 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.579+0000 7f2be8c9a640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:04.581 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.579+0000 7f2be8c9a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:20:04.581 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.579+0000 7f2be8c9a640 10 monclient(hunting): picked mon.noname-c con 0x7f2be40b4670 addr v1:172.21.15.6:6790/0 2024-03-25T19:20:04.581 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.579+0000 7f2be8c9a640 10 monclient(hunting): picked mon.noname-a con 0x7f2be40b4ff0 addr v1:172.21.15.6:6789/0 2024-03-25T19:20:04.581 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.579+0000 7f2be8c9a640 10 monclient(hunting): picked mon.noname-b con 0x7f2be413c480 addr v1:172.21.15.192:6789/0 2024-03-25T19:20:04.581 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.579+0000 7f2be8c9a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2be40b0160 con 0x7f2be40b4ff0 2024-03-25T19:20:04.582 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.579+0000 7f2be8c9a640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2be40b0440 con 0x7f2be413c480 2024-03-25T19:20:04.582 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.579+0000 7f2be8c9a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2be40b0720 con 0x7f2be40b4670 2024-03-25T19:20:04.582 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.579+0000 7f2be8c9a640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:04.582 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.579+0000 7f2be8c9a640 10 monclient(hunting): authenticate will time out at 6308.819176s 2024-03-25T19:20:04.590 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.590+0000 7f2be2d9d640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f2be413c480 0x7f2be40aeb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:45086/0 (socket says 172.21.15.6:45086) 2024-03-25T19:20:04.590 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.590+0000 7f2be2d9d640 1 -- 172.21.15.6:0/4082951249 learned_addr learned my addr 172.21.15.6:0/4082951249 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:20:04.591 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.591+0000 7f2bd2ffd640 10 client.?.objecter ms_handle_connect 0x7f2be413c480 2024-03-25T19:20:04.591 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.591+0000 7f2bd2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:04.601 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.601+0000 7f2bd2ffd640 10 client.?.objecter ms_handle_connect 0x7f2be40b4ff0 2024-03-25T19:20:04.601 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.601+0000 7f2bd2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:04.606 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.606+0000 7f2bd2ffd640 10 client.?.objecter ms_handle_connect 0x7f2be40b4670 2024-03-25T19:20:04.606 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.606+0000 7f2bd2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:04.607 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.606+0000 7f2bd2ffd640 1 -- 172.21.15.6:0/4082951249 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2001176078 0 0) 0x7f2be40b0440 con 0x7f2be413c480 2024-03-25T19:20:04.607 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.606+0000 7f2bd2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:20:04.607 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.606+0000 7f2bd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:04.607 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.606+0000 7f2bd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:04.607 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.606+0000 7f2bd2ffd640 10 monclient(hunting): my global_id is 4498 2024-03-25T19:20:04.607 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.606+0000 7f2bd2ffd640 1 -- 172.21.15.6:0/4082951249 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bc4004eb0 con 0x7f2be413c480 2024-03-25T19:20:04.620 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.620+0000 7f2bd2ffd640 1 -- 172.21.15.6:0/4082951249 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2054621734 0 0) 0x7f2be40b0160 con 0x7f2be40b4ff0 2024-03-25T19:20:04.620 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.620+0000 7f2bd2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:20:04.621 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.620+0000 7f2bd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:04.621 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.620+0000 7f2bd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:04.621 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.620+0000 7f2bd2ffd640 10 monclient(hunting): my global_id is 4494 2024-03-25T19:20:04.621 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.620+0000 7f2bd2ffd640 1 -- 172.21.15.6:0/4082951249 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2be40b0440 con 0x7f2be40b4ff0 2024-03-25T19:20:04.630 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.630+0000 7f2bd2ffd640 1 -- 172.21.15.6:0/4082951249 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1422642121 0 0) 0x7f2bc4004eb0 con 0x7f2be413c480 2024-03-25T19:20:04.631 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.630+0000 7f2bd2ffd640 1 -- 172.21.15.6:0/4082951249 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2be40b0160 con 0x7f2be413c480 2024-03-25T19:20:04.631 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.630+0000 7f2bd2ffd640 1 -- 172.21.15.6:0/4082951249 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f2bdc001ec0 con 0x7f2be413c480 2024-03-25T19:20:04.631 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.630+0000 7f2bd2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:04.631 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.630+0000 7f2bd2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:20:04.631 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.630+0000 7f2bd2ffd640 10 monclient(hunting): dump: 2024-03-25T19:20:04.631 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:20:04.631 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:04.631 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:04.631 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:04.631 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:04.631 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:04.631 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:20:04.631 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:20:04.631 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:20:04.632 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:04.632 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.630+0000 7f2bd2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:20:04.632 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.631+0000 7f2bd2ffd640 1 -- 172.21.15.6:0/4082951249 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2163048902 0 0) 0x7f2be40b0720 con 0x7f2be40b4670 2024-03-25T19:20:04.632 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.631+0000 7f2bd2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:20:04.632 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.631+0000 7f2bd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:04.632 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.631+0000 7f2bd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:04.632 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.631+0000 7f2bd2ffd640 10 monclient(hunting): my global_id is 4493 2024-03-25T19:20:04.632 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.631+0000 7f2bd2ffd640 1 -- 172.21.15.6:0/4082951249 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bc4004eb0 con 0x7f2be40b4670 2024-03-25T19:20:04.639 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.639+0000 7f2bd2ffd640 1 -- 172.21.15.6:0/4082951249 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2090880226 0 0) 0x7f2be40b0440 con 0x7f2be40b4ff0 2024-03-25T19:20:04.639 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.639+0000 7f2bd2ffd640 1 -- 172.21.15.6:0/4082951249 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2be40b0720 con 0x7f2be40b4ff0 2024-03-25T19:20:04.639 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.639+0000 7f2bd2ffd640 1 -- 172.21.15.6:0/4082951249 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f2bd80018b0 con 0x7f2be40b4ff0 2024-03-25T19:20:04.639 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.639+0000 7f2bd2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:04.639 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.639+0000 7f2bd2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:20:04.639 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.639+0000 7f2bd2ffd640 10 monclient(hunting): dump: 2024-03-25T19:20:04.639 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:20:04.639 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:04.639 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:04.639 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:04.639 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:04.640 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:04.640 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:20:04.640 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:20:04.640 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:20:04.640 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:04.642 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.641+0000 7f2bd2ffd640 1 -- 172.21.15.6:0/4082951249 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 55206215 0 0) 0x7f2be40b0160 con 0x7f2be413c480 2024-03-25T19:20:04.642 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.641+0000 7f2bd2ffd640 1 -- 172.21.15.6:0/4082951249 >> v1:172.21.15.6:6790/0 conn(0x7f2be40b4670 legacy=0x7f2be40b0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:04.642 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.641+0000 7f2bd2ffd640 1 -- 172.21.15.6:0/4082951249 >> v1:172.21.15.6:6789/0 conn(0x7f2be40b4ff0 legacy=0x7f2be40ae430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:04.642 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.641+0000 7f2bd2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:20:04.642 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.641+0000 7f2bd2ffd640 1 monclient: found mon.b 2024-03-25T19:20:04.642 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.641+0000 7f2bd2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:04.642 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.641+0000 7f2bd2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:20:04.642 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.641+0000 7f2bd2ffd640 1 -- 172.21.15.6:0/4082951249 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be40b15c0 con 0x7f2be413c480 2024-03-25T19:20:04.642 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.641+0000 7f2bd2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:20:04.643 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.641+0000 7f2bd2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:20:04.643 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.641+0000 7f2bd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:04.643 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.641+0000 7f2be8c9a640 5 monclient: authenticate success, global_id 4498 2024-03-25T19:20:04.643 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.641+0000 7f2be8c9a640 10 monclient: _renew_subs 2024-03-25T19:20:04.643 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.641+0000 7f2be8c9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:20:04.643 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.641+0000 7f2be8c9a640 1 -- 172.21.15.6:0/4082951249 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2be41e35a0 con 0x7f2be413c480 2024-03-25T19:20:04.643 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.641+0000 7f2be8c9a640 10 client.4498.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:20:04.643 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.641+0000 7f2be8c9a640 10 monclient: _renew_subs 2024-03-25T19:20:04.643 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.641+0000 7f2be8c9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:20:04.643 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.641+0000 7f2be8c9a640 1 -- 172.21.15.6:0/4082951249 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2be41e3910 con 0x7f2be413c480 2024-03-25T19:20:04.643 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.641+0000 7f2be8c9a640 1 librados: init done 2024-03-25T19:20:04.644 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.642+0000 7f2be8c9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:04.644 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.642+0000 7f2be8c9a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:04.644 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.642+0000 7f2be8c9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:20:04.644 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.642+0000 7f2be8c9a640 1 -- 172.21.15.6:0/4082951249 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2be40ae150 con 0x7f2be413c480 2024-03-25T19:20:04.919 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.919+0000 7f2bd2ffd640 1 -- 172.21.15.6:0/4082951249 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2bdc004e10 con 0x7f2be413c480 2024-03-25T19:20:04.919 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.919+0000 7f2bd2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:04.919 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.919+0000 7f2bd2ffd640 1 -- 172.21.15.6:0/4082951249 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f2bdc0015d0 con 0x7f2be413c480 2024-03-25T19:20:04.919 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.919+0000 7f2bd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:04.920 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.919+0000 7f2bd2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:20:04.920 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.919+0000 7f2bd2ffd640 10 monclient: dump: 2024-03-25T19:20:04.920 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:20:04.920 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:04.920 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:04.920 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:04.920 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:04.920 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:04.920 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:20:04.920 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:20:04.920 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:20:04.920 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:04.920 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.919+0000 7f2be0d99640 4 set_mon_vals no callback set 2024-03-25T19:20:04.921 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.920+0000 7f2bd2ffd640 1 -- 172.21.15.6:0/4082951249 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f2bdc01adf0 con 0x7f2be413c480 2024-03-25T19:20:04.921 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.920+0000 7f2be259c640 10 monclient: get_auth_request con 0x7f2bc406df20 auth_method 0 2024-03-25T19:20:04.921 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.921+0000 7f2bd2ffd640 10 client.4498.objecter ms_handle_connect 0x7f2bc406df20 2024-03-25T19:20:04.924 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.924+0000 7f2bd2ffd640 1 -- 172.21.15.6:0/4082951249 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f2bdc054a30 con 0x7f2be413c480 2024-03-25T19:20:04.924 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.924+0000 7f2bd2ffd640 10 client.4498.objecter ms_dispatch 0x7f2be4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:20:04.924 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.924+0000 7f2bd2ffd640 3 client.4498.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:20:04.924 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.924+0000 7f2bd2ffd640 3 client.4498.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:20:04.924 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.924+0000 7f2bd2ffd640 20 client.4498.objecter dump_active .. 0 homeless 2024-03-25T19:20:04.925 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.924+0000 7f2bd2ffd640 1 -- 172.21.15.6:0/4082951249 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2bdc050370 con 0x7f2be413c480 2024-03-25T19:20:04.925 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.924+0000 7f2bd2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:04.925 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:04.924+0000 7f2bd2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:20:05.161 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:05.160+0000 7f2be8c9a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:05.161 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:05.160+0000 7f2be8c9a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:05.161 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:05.160+0000 7f2be8c9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:20:05.161 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:05.160+0000 7f2be8c9a640 1 -- 172.21.15.6:0/4082951249 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2be406dbd0 con 0x7f2be413c480 2024-03-25T19:20:05.197 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:05.197+0000 7f2bd2ffd640 1 -- 172.21.15.6:0/4082951249 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f2bdc052470 con 0x7f2be413c480 2024-03-25T19:20:05.197 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:05.197+0000 7f2bd2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:05.197 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:05.197+0000 7f2bd2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:20:05.197 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:20:05.197 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:20:05.200 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:05.199+0000 7f2be8c9a640 10 librados: watch_flush enter 2024-03-25T19:20:05.200 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:05.199+0000 7f2be8c9a640 10 librados: watch_flush exit 2024-03-25T19:20:05.200 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:05.199+0000 7f2be8c9a640 20 client.4498.objecter shutdown clearing up homeless session... 2024-03-25T19:20:05.200 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:05.199+0000 7f2be8c9a640 10 client.4498.objecter successfully canceled tick 2024-03-25T19:20:05.200 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:05.200+0000 7f2be8c9a640 1 -- 172.21.15.6:0/4082951249 >> v1:172.21.15.6:6800/2936652650 conn(0x7f2bc406df20 legacy=0x7f2bc40703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:05.201 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:05.200+0000 7f2be8c9a640 10 monclient: shutdown 2024-03-25T19:20:05.201 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:05.200+0000 7f2be8c9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:05.201 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:05.200+0000 7f2be8c9a640 1 -- 172.21.15.6:0/4082951249 >> v1:172.21.15.192:6789/0 conn(0x7f2be413c480 legacy=0x7f2be40aeb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:05.201 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:05.200+0000 7f2be8c9a640 1 -- 172.21.15.6:0/4082951249 shutdown_connections 2024-03-25T19:20:05.201 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:05.200+0000 7f2be8c9a640 1 -- 172.21.15.6:0/4082951249 >> 172.21.15.6:0/4082951249 conn(0x7f2be40617a0 msgr2=0x7f2be4145420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:05.201 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:05.200+0000 7f2be8c9a640 1 -- 172.21.15.6:0/4082951249 shutdown_connections 2024-03-25T19:20:05.201 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:05.200+0000 7f2be8c9a640 1 -- 172.21.15.6:0/4082951249 wait complete. 2024-03-25T19:20:05.201 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:05.200+0000 7f2be8c9a640 1 librados: shutdown 2024-03-25T19:20:05.213 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:20:08.540 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:06:25.887 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:08.596 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:06:25.943 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:08.623 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:06:25.970 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:08.637 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:06:25.955 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:08.646 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:06:25.964 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:08.654 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:06:26.001 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:08.660 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:06:26.007 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:08.677 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:06:25.994 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:08.700 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:06:26.047 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:12.018 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:20:12.131 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.131+0000 7f197cf48640 10 monclient: get_monmap_and_config 2024-03-25T19:20:12.131 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.131+0000 7f197cf48640 10 monclient: build_initial_monmap 2024-03-25T19:20:12.131 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.131+0000 7f197cf48640 10 monclient: monmap: 2024-03-25T19:20:12.131 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:20:12.131 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:12.131 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:20:12.132392+0000 2024-03-25T19:20:12.131 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:20:12.132392+0000 2024-03-25T19:20:12.132 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:12.132 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:12.132 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:20:12.132 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:20:12.132 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:20:12.132 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:12.133 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.132+0000 7f197cf48640 1 Processor -- start 2024-03-25T19:20:12.133 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.132+0000 7f197cf48640 1 -- start start 2024-03-25T19:20:12.133 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.132+0000 7f197cf48640 10 monclient: init 2024-03-25T19:20:12.133 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.132+0000 7f197cf48640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:12.133 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.132+0000 7f197cf48640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:20:12.133 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.132+0000 7f197cf48640 10 monclient(hunting): picked mon.noname-a con 0x7f19780b4e70 addr v1:172.21.15.6:6789/0 2024-03-25T19:20:12.133 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.132+0000 7f197cf48640 10 monclient(hunting): picked mon.noname-b con 0x7f19780b44b0 addr v1:172.21.15.6:6790/0 2024-03-25T19:20:12.133 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.132+0000 7f197cf48640 10 monclient(hunting): picked mon.noname-c con 0x7f19780b1b00 addr v1:172.21.15.192:6789/0 2024-03-25T19:20:12.133 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.132+0000 7f197cf48640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19780a96f0 con 0x7f19780b4e70 2024-03-25T19:20:12.133 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.132+0000 7f197cf48640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19780ab0e0 con 0x7f19780b1b00 2024-03-25T19:20:12.134 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.132+0000 7f197cf48640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1978145f70 con 0x7f19780b44b0 2024-03-25T19:20:12.134 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.132+0000 7f197cf48640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:12.134 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.132+0000 7f197cf48640 10 monclient(hunting): authenticate will time out at 6316.372632s 2024-03-25T19:20:12.144 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.144+0000 7f197659c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f19780b4e70 0x7f197813f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50366/0 (socket says 172.21.15.6:50366) 2024-03-25T19:20:12.144 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.144+0000 7f197659c640 1 -- 172.21.15.6:0/2564643397 learned_addr learned my addr 172.21.15.6:0/2564643397 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:20:12.170 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.170+0000 7f1975d9b640 1 -- 172.21.15.6:0/2564643397 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2715826279 0 0) 0x7f19780ab0e0 con 0x7f19780b1b00 2024-03-25T19:20:12.170 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.170+0000 7f1975d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:20:12.170 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.170+0000 7f1975d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:12.170 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.170+0000 7f1975d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:12.170 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.170+0000 7f1975d9b640 10 monclient(hunting): my global_id is 4501 2024-03-25T19:20:12.171 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.170+0000 7f1975d9b640 1 -- 172.21.15.6:0/2564643397 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1964003520 con 0x7f19780b1b00 2024-03-25T19:20:12.171 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.171+0000 7f1975d9b640 1 -- 172.21.15.6:0/2564643397 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1056978583 0 0) 0x7f19780a96f0 con 0x7f19780b4e70 2024-03-25T19:20:12.171 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.171+0000 7f1975d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:20:12.171 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.171+0000 7f1975d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:12.172 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.171+0000 7f1975d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:12.172 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.171+0000 7f1975d9b640 10 monclient(hunting): my global_id is 4497 2024-03-25T19:20:12.172 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.171+0000 7f1975d9b640 1 -- 172.21.15.6:0/2564643397 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19780ab0e0 con 0x7f19780b4e70 2024-03-25T19:20:12.176 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.176+0000 7f1975d9b640 1 -- 172.21.15.6:0/2564643397 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3558660871 0 0) 0x7f1978145f70 con 0x7f19780b44b0 2024-03-25T19:20:12.176 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.176+0000 7f1975d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:20:12.176 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.176+0000 7f1975d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:12.176 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.176+0000 7f1975d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:12.176 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.176+0000 7f1975d9b640 10 monclient(hunting): my global_id is 4496 2024-03-25T19:20:12.177 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.176+0000 7f1975d9b640 1 -- 172.21.15.6:0/2564643397 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19780a96f0 con 0x7f19780b44b0 2024-03-25T19:20:12.191 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.190+0000 7f1975d9b640 1 -- 172.21.15.6:0/2564643397 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2315746650 0 0) 0x7f19780ab0e0 con 0x7f19780b4e70 2024-03-25T19:20:12.191 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.190+0000 7f1975d9b640 1 -- 172.21.15.6:0/2564643397 >> v1:172.21.15.6:6790/0 conn(0x7f19780b44b0 legacy=0x7f1978145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:12.191 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.190+0000 7f1975d9b640 1 -- 172.21.15.6:0/2564643397 >> v1:172.21.15.192:6789/0 conn(0x7f19780b1b00 legacy=0x7f197814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:12.191 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.190+0000 7f1975d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:20:12.191 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.190+0000 7f1975d9b640 1 monclient: found mon.noname-a 2024-03-25T19:20:12.191 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.190+0000 7f1975d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:12.191 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.190+0000 7f1975d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-03-25T19:20:12.191 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.190+0000 7f1975d9b640 1 -- 172.21.15.6:0/2564643397 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19780a9b80 con 0x7f19780b4e70 2024-03-25T19:20:12.191 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.190+0000 7f1975d9b640 10 monclient: _finish_auth 0 2024-03-25T19:20:12.191 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.190+0000 7f1975d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:20:12.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.190+0000 7f1975d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:12.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.190+0000 7f1975d9b640 1 -- 172.21.15.6:0/2564643397 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f1968001670 con 0x7f19780b4e70 2024-03-25T19:20:12.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.190+0000 7f1975d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:12.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.191+0000 7f1975d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:20:12.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.191+0000 7f1975d9b640 10 monclient: dump: 2024-03-25T19:20:12.192 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:20:12.192 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:12.192 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:12.192 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:12.192 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:12.192 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:12.192 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:20:12.192 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:20:12.192 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:20:12.192 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:12.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.191+0000 7f197cf48640 5 monclient: authenticate success, global_id 4497 2024-03-25T19:20:12.193 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.191+0000 7f197cf48640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:12.218 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.218+0000 7f1975d9b640 1 -- 172.21.15.6:0/2564643397 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19680017d0 con 0x7f19780b4e70 2024-03-25T19:20:12.218 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.218+0000 7f1975d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:12.218 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.218+0000 7f1975d9b640 1 -- 172.21.15.6:0/2564643397 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f1968001be0 con 0x7f19780b4e70 2024-03-25T19:20:12.218 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.218+0000 7f197cf48640 10 monclient: get_monmap_and_config success 2024-03-25T19:20:12.218 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.218+0000 7f197cf48640 4 set_mon_vals no callback set 2024-03-25T19:20:12.218 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.218+0000 7f1975d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:12.218 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.218+0000 7f1975d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:20:12.219 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.218+0000 7f1975d9b640 10 monclient: dump: 2024-03-25T19:20:12.219 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:20:12.219 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:12.219 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:12.219 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:12.219 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:12.219 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:12.219 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:20:12.219 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:20:12.219 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:20:12.219 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:12.220 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.219+0000 7f197cf48640 10 monclient: shutdown 2024-03-25T19:20:12.220 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.219+0000 7f197cf48640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:12.221 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.219+0000 7f197cf48640 1 -- 172.21.15.6:0/2564643397 >> v1:172.21.15.6:6789/0 conn(0x7f19780b4e70 legacy=0x7f197813f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:12.221 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.219+0000 7f197cf48640 1 -- 172.21.15.6:0/2564643397 shutdown_connections 2024-03-25T19:20:12.221 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.219+0000 7f197cf48640 1 -- 172.21.15.6:0/2564643397 >> 172.21.15.6:0/2564643397 conn(0x7f19780617a0 msgr2=0x7f1978061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:12.221 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.219+0000 7f197cf48640 1 -- 172.21.15.6:0/2564643397 shutdown_connections 2024-03-25T19:20:12.221 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.219+0000 7f197cf48640 1 -- 172.21.15.6:0/2564643397 wait complete. 2024-03-25T19:20:12.221 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.220+0000 7f197cf48640 10 monclient: build_initial_monmap 2024-03-25T19:20:12.221 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.220+0000 7f197cf48640 10 monclient: monmap: 2024-03-25T19:20:12.221 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:20:12.221 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:20:12.221 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:20:12.221 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:20:12.221 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:12.221 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:20:12.221 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:20:12.222 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:20:12.222 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:20:12.222 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:12.222 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.220+0000 7f197cf48640 1 librados: starting msgr at 2024-03-25T19:20:12.222 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.220+0000 7f197cf48640 1 librados: starting objecter 2024-03-25T19:20:12.222 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.220+0000 7f197cf48640 1 Processor -- start 2024-03-25T19:20:12.222 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.220+0000 7f197cf48640 1 -- start start 2024-03-25T19:20:12.222 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.220+0000 7f197cf48640 1 librados: setting wanted keys 2024-03-25T19:20:12.222 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.220+0000 7f197cf48640 1 librados: calling monclient init 2024-03-25T19:20:12.222 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.220+0000 7f197cf48640 10 monclient: init 2024-03-25T19:20:12.223 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.221+0000 7f197cf48640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:12.223 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.221+0000 7f197cf48640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:20:12.223 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.221+0000 7f197cf48640 10 monclient(hunting): picked mon.noname-a con 0x7f19780b1b00 addr v1:172.21.15.6:6789/0 2024-03-25T19:20:12.223 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.221+0000 7f197cf48640 10 monclient(hunting): picked mon.noname-c con 0x7f19780b44b0 addr v1:172.21.15.6:6790/0 2024-03-25T19:20:12.223 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.221+0000 7f197cf48640 10 monclient(hunting): picked mon.noname-b con 0x7f19780b4e70 addr v1:172.21.15.192:6789/0 2024-03-25T19:20:12.223 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.221+0000 7f197cf48640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19781d7750 con 0x7f19780b1b00 2024-03-25T19:20:12.223 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.221+0000 7f197cf48640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19781d8390 con 0x7f19780b4e70 2024-03-25T19:20:12.223 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.221+0000 7f197cf48640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19781d8520 con 0x7f19780b44b0 2024-03-25T19:20:12.224 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.221+0000 7f197cf48640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:12.224 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.221+0000 7f197cf48640 10 monclient(hunting): authenticate will time out at 6316.461294s 2024-03-25T19:20:12.236 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.236+0000 7f197659c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f19780b1b00 0x7f19781d7c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50414/0 (socket says 172.21.15.6:50414) 2024-03-25T19:20:12.236 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.236+0000 7f197659c640 1 -- 172.21.15.6:0/2182513963 learned_addr learned my addr 172.21.15.6:0/2182513963 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:20:12.240 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.240+0000 7f196effd640 10 client.?.objecter ms_handle_connect 0x7f19780b1b00 2024-03-25T19:20:12.240 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.240+0000 7f196effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:12.243 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.243+0000 7f196effd640 10 client.?.objecter ms_handle_connect 0x7f19780b44b0 2024-03-25T19:20:12.243 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.243+0000 7f196effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:12.243 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.243+0000 7f196effd640 10 client.?.objecter ms_handle_connect 0x7f19780b4e70 2024-03-25T19:20:12.243 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.243+0000 7f196effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:12.248 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.248+0000 7f196effd640 1 -- 172.21.15.6:0/2182513963 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 41880618 0 0) 0x7f19781d7750 con 0x7f19780b1b00 2024-03-25T19:20:12.248 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.248+0000 7f196effd640 10 monclient(hunting): _negotiate 2024-03-25T19:20:12.248 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.248+0000 7f196effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:12.248 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.248+0000 7f196effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:12.248 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.248+0000 7f196effd640 10 monclient(hunting): my global_id is 4500 2024-03-25T19:20:12.248 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.248+0000 7f196effd640 1 -- 172.21.15.6:0/2182513963 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1958004eb0 con 0x7f19780b1b00 2024-03-25T19:20:12.249 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.249+0000 7f196effd640 1 -- 172.21.15.6:0/2182513963 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4036020819 0 0) 0x7f19781d8520 con 0x7f19780b44b0 2024-03-25T19:20:12.249 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.249+0000 7f196effd640 10 monclient(hunting): _negotiate 2024-03-25T19:20:12.249 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.249+0000 7f196effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:12.249 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.249+0000 7f196effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:12.249 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.249+0000 7f196effd640 10 monclient(hunting): my global_id is 4499 2024-03-25T19:20:12.249 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.249+0000 7f196effd640 1 -- 172.21.15.6:0/2182513963 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19781d7750 con 0x7f19780b44b0 2024-03-25T19:20:12.250 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.250+0000 7f196effd640 1 -- 172.21.15.6:0/2182513963 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1352195242 0 0) 0x7f19781d8390 con 0x7f19780b4e70 2024-03-25T19:20:12.250 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.250+0000 7f196effd640 10 monclient(hunting): _negotiate 2024-03-25T19:20:12.250 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.250+0000 7f196effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:12.250 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.250+0000 7f196effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:12.250 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.250+0000 7f196effd640 10 monclient(hunting): my global_id is 4504 2024-03-25T19:20:12.250 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.250+0000 7f196effd640 1 -- 172.21.15.6:0/2182513963 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19781d8520 con 0x7f19780b4e70 2024-03-25T19:20:12.257 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.257+0000 7f196effd640 1 -- 172.21.15.6:0/2182513963 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1297174955 0 0) 0x7f19781d7750 con 0x7f19780b44b0 2024-03-25T19:20:12.257 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.257+0000 7f196effd640 1 -- 172.21.15.6:0/2182513963 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19781d8390 con 0x7f19780b44b0 2024-03-25T19:20:12.257 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.257+0000 7f196effd640 1 -- 172.21.15.6:0/2182513963 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f1960001870 con 0x7f19780b44b0 2024-03-25T19:20:12.257 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.257+0000 7f196effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:12.257 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.257+0000 7f196effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:20:12.257 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.257+0000 7f196effd640 10 monclient(hunting): dump: 2024-03-25T19:20:12.257 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:20:12.257 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:12.257 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:12.257 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:12.257 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:12.258 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:12.258 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:20:12.258 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:20:12.258 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:20:12.258 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:12.258 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.257+0000 7f196effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:20:12.261 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.261+0000 7f196effd640 1 -- 172.21.15.6:0/2182513963 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 717904995 0 0) 0x7f19781d8390 con 0x7f19780b44b0 2024-03-25T19:20:12.261 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.261+0000 7f196effd640 1 -- 172.21.15.6:0/2182513963 >> v1:172.21.15.192:6789/0 conn(0x7f19780b4e70 legacy=0x7f19781daff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:12.261 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.261+0000 7f196effd640 1 -- 172.21.15.6:0/2182513963 >> v1:172.21.15.6:6789/0 conn(0x7f19780b1b00 legacy=0x7f19781d7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:12.261 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.261+0000 7f196effd640 10 monclient: _finish_hunting 0 2024-03-25T19:20:12.261 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.261+0000 7f196effd640 1 monclient: found mon.c 2024-03-25T19:20:12.261 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.261+0000 7f196effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:12.262 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.261+0000 7f196effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:20:12.262 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.261+0000 7f196effd640 1 -- 172.21.15.6:0/2182513963 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19781d86b0 con 0x7f19780b44b0 2024-03-25T19:20:12.262 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.261+0000 7f196effd640 10 monclient: _finish_auth 0 2024-03-25T19:20:12.262 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.261+0000 7f196effd640 10 monclient: _check_auth_tickets 2024-03-25T19:20:12.262 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.261+0000 7f196effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:12.262 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.261+0000 7f197cf48640 5 monclient: authenticate success, global_id 4499 2024-03-25T19:20:12.262 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.261+0000 7f197cf48640 10 monclient: _renew_subs 2024-03-25T19:20:12.262 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.261+0000 7f197cf48640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:20:12.262 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.261+0000 7f197cf48640 1 -- 172.21.15.6:0/2182513963 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1978145470 con 0x7f19780b44b0 2024-03-25T19:20:12.262 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.261+0000 7f197cf48640 10 client.4499.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:20:12.262 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.261+0000 7f197cf48640 10 monclient: _renew_subs 2024-03-25T19:20:12.262 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.261+0000 7f197cf48640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:20:12.262 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.261+0000 7f197cf48640 1 -- 172.21.15.6:0/2182513963 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f197813b790 con 0x7f19780b44b0 2024-03-25T19:20:12.262 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.261+0000 7f197cf48640 1 librados: init done 2024-03-25T19:20:12.263 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.262+0000 7f197cf48640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:12.263 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.262+0000 7f197cf48640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:12.263 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.262+0000 7f197cf48640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:20:12.263 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.262+0000 7f197cf48640 1 -- 172.21.15.6:0/2182513963 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19781dac50 con 0x7f19780b44b0 2024-03-25T19:20:12.597 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.596+0000 7f196effd640 1 -- 172.21.15.6:0/2182513963 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1960003610 con 0x7f19780b44b0 2024-03-25T19:20:12.607 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.597+0000 7f196effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:12.607 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.597+0000 7f196effd640 1 -- 172.21.15.6:0/2182513963 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f1960001d80 con 0x7f19780b44b0 2024-03-25T19:20:12.608 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.597+0000 7f196effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:12.608 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.597+0000 7f196effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:20:12.608 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.597+0000 7f196effd640 10 monclient: dump: 2024-03-25T19:20:12.608 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:20:12.608 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:12.608 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:12.608 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:12.608 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:12.608 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:12.608 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:20:12.608 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:20:12.609 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:20:12.609 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:12.609 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.597+0000 7f197559a640 4 set_mon_vals no callback set 2024-03-25T19:20:12.610 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.597+0000 7f196effd640 1 -- 172.21.15.6:0/2182513963 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f1960003b20 con 0x7f19780b44b0 2024-03-25T19:20:12.610 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.598+0000 7f197659c640 10 monclient: get_auth_request con 0x7f195806de20 auth_method 0 2024-03-25T19:20:12.610 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.598+0000 7f196effd640 10 client.4499.objecter ms_handle_connect 0x7f195806de20 2024-03-25T19:20:12.610 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.603+0000 7f196effd640 1 -- 172.21.15.6:0/2182513963 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f1960053500 con 0x7f19780b44b0 2024-03-25T19:20:12.610 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.603+0000 7f196effd640 10 client.4499.objecter ms_dispatch 0x7f1978000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:20:12.610 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.604+0000 7f196effd640 3 client.4499.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:20:12.610 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.604+0000 7f196effd640 3 client.4499.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:20:12.610 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.604+0000 7f196effd640 20 client.4499.objecter dump_active .. 0 homeless 2024-03-25T19:20:12.610 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.604+0000 7f196effd640 1 -- 172.21.15.6:0/2182513963 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f196004f010 con 0x7f19780b44b0 2024-03-25T19:20:12.610 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.604+0000 7f196effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:12.610 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.604+0000 7f196effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:20:12.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.841+0000 7f197cf48640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:12.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.841+0000 7f197cf48640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:12.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.841+0000 7f197cf48640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:20:12.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.841+0000 7f197cf48640 1 -- 172.21.15.6:0/2182513963 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1978145810 con 0x7f19780b44b0 2024-03-25T19:20:12.871 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.871+0000 7f196effd640 1 -- 172.21.15.6:0/2182513963 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f1960050f40 con 0x7f19780b44b0 2024-03-25T19:20:12.872 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.871+0000 7f196effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:12.872 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.871+0000 7f196effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:20:12.872 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:20:12.872 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:20:12.874 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.874+0000 7f197cf48640 10 librados: watch_flush enter 2024-03-25T19:20:12.874 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.874+0000 7f197cf48640 10 librados: watch_flush exit 2024-03-25T19:20:12.874 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.874+0000 7f197cf48640 20 client.4499.objecter shutdown clearing up homeless session... 2024-03-25T19:20:12.875 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.874+0000 7f197cf48640 10 client.4499.objecter successfully canceled tick 2024-03-25T19:20:12.875 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.874+0000 7f197cf48640 1 -- 172.21.15.6:0/2182513963 >> v1:172.21.15.6:6800/2936652650 conn(0x7f195806de20 legacy=0x7f19580702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:12.875 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.874+0000 7f197cf48640 10 monclient: shutdown 2024-03-25T19:20:12.876 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.874+0000 7f197cf48640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:12.876 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.874+0000 7f197cf48640 1 -- 172.21.15.6:0/2182513963 >> v1:172.21.15.6:6790/0 conn(0x7f19780b44b0 legacy=0x7f197813aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:12.876 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.875+0000 7f197cf48640 1 -- 172.21.15.6:0/2182513963 shutdown_connections 2024-03-25T19:20:12.876 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.875+0000 7f197cf48640 1 -- 172.21.15.6:0/2182513963 >> 172.21.15.6:0/2182513963 conn(0x7f19780617a0 msgr2=0x7f19780b59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:12.876 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.875+0000 7f197cf48640 1 -- 172.21.15.6:0/2182513963 shutdown_connections 2024-03-25T19:20:12.876 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.875+0000 7f197cf48640 1 -- 172.21.15.6:0/2182513963 wait complete. 2024-03-25T19:20:12.876 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:12.875+0000 7f197cf48640 1 librados: shutdown 2024-03-25T19:20:12.888 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:20:19.693 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:20:19.813 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.813+0000 7f67d3915640 10 monclient: get_monmap_and_config 2024-03-25T19:20:19.813 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.813+0000 7f67d3915640 10 monclient: build_initial_monmap 2024-03-25T19:20:19.813 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.813+0000 7f67d3915640 10 monclient: monmap: 2024-03-25T19:20:19.813 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:20:19.813 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:19.813 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:20:19.814190+0000 2024-03-25T19:20:19.813 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:20:19.814190+0000 2024-03-25T19:20:19.813 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:19.813 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:19.813 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:20:19.814 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:20:19.814 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:20:19.814 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:19.815 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.813+0000 7f67d3915640 1 Processor -- start 2024-03-25T19:20:19.815 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.814+0000 7f67d3915640 1 -- start start 2024-03-25T19:20:19.815 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.814+0000 7f67d3915640 10 monclient: init 2024-03-25T19:20:19.815 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.814+0000 7f67d3915640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:19.815 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.814+0000 7f67d3915640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:20:19.815 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.814+0000 7f67d3915640 10 monclient(hunting): picked mon.noname-c con 0x7f67cc0b4ff0 addr v1:172.21.15.192:6789/0 2024-03-25T19:20:19.815 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.814+0000 7f67d3915640 10 monclient(hunting): picked mon.noname-b con 0x7f67cc0b4670 addr v1:172.21.15.6:6790/0 2024-03-25T19:20:19.815 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.814+0000 7f67d3915640 10 monclient(hunting): picked mon.noname-a con 0x7f67cc13c480 addr v1:172.21.15.6:6789/0 2024-03-25T19:20:19.815 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.814+0000 7f67d3915640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67cc0ab0e0 con 0x7f67cc13c480 2024-03-25T19:20:19.815 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.814+0000 7f67d3915640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67cc0ab3c0 con 0x7f67cc0b4ff0 2024-03-25T19:20:19.815 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.814+0000 7f67d3915640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67cc0a9f30 con 0x7f67cc0b4670 2024-03-25T19:20:19.815 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.814+0000 7f67d3915640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:19.816 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.814+0000 7f67d3915640 10 monclient(hunting): authenticate will time out at 6324.054629s 2024-03-25T19:20:19.819 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.820+0000 7f67d16b1640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f67cc0b4ff0 0x7f67cc13bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:53940/0 (socket says 172.21.15.6:53940) 2024-03-25T19:20:19.820 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.820+0000 7f67d16b1640 1 -- 172.21.15.6:0/1624931945 learned_addr learned my addr 172.21.15.6:0/1624931945 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:20:19.827 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.826+0000 7f67c3fff640 1 -- 172.21.15.6:0/1624931945 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 96220349 0 0) 0x7f67cc0ab3c0 con 0x7f67cc0b4ff0 2024-03-25T19:20:19.827 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.827+0000 7f67c3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:20:19.827 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.827+0000 7f67c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:19.827 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.827+0000 7f67c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:19.827 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.827+0000 7f67c3fff640 10 monclient(hunting): my global_id is 4507 2024-03-25T19:20:19.827 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.827+0000 7f67c3fff640 1 -- 172.21.15.6:0/1624931945 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67b8003520 con 0x7f67cc0b4ff0 2024-03-25T19:20:19.838 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.838+0000 7f67c3fff640 1 -- 172.21.15.6:0/1624931945 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4028024085 0 0) 0x7f67cc0a9f30 con 0x7f67cc0b4670 2024-03-25T19:20:19.838 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.838+0000 7f67c3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:20:19.838 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.838+0000 7f67c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:19.838 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.838+0000 7f67c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:19.838 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.838+0000 7f67c3fff640 10 monclient(hunting): my global_id is 4502 2024-03-25T19:20:19.838 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.838+0000 7f67c3fff640 1 -- 172.21.15.6:0/1624931945 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67cc0ab3c0 con 0x7f67cc0b4670 2024-03-25T19:20:19.840 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.840+0000 7f67c3fff640 1 -- 172.21.15.6:0/1624931945 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2348104074 0 0) 0x7f67cc0ab0e0 con 0x7f67cc13c480 2024-03-25T19:20:19.840 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.840+0000 7f67c3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:20:19.840 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.840+0000 7f67c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:19.840 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.840+0000 7f67c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:19.840 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.840+0000 7f67c3fff640 10 monclient(hunting): my global_id is 4503 2024-03-25T19:20:19.840 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.840+0000 7f67c3fff640 1 -- 172.21.15.6:0/1624931945 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67cc0a9f30 con 0x7f67cc13c480 2024-03-25T19:20:19.840 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.840+0000 7f67c3fff640 1 -- 172.21.15.6:0/1624931945 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1136764959 0 0) 0x7f67b8003520 con 0x7f67cc0b4ff0 2024-03-25T19:20:19.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.840+0000 7f67c3fff640 1 -- 172.21.15.6:0/1624931945 >> v1:172.21.15.6:6790/0 conn(0x7f67cc0b4670 legacy=0x7f67cc1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:19.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.840+0000 7f67c3fff640 1 -- 172.21.15.6:0/1624931945 >> v1:172.21.15.6:6789/0 conn(0x7f67cc13c480 legacy=0x7f67cc14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:19.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.840+0000 7f67c3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:20:19.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.840+0000 7f67c3fff640 1 monclient: found mon.noname-c 2024-03-25T19:20:19.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.840+0000 7f67c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:19.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.840+0000 7f67c3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:20:19.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.840+0000 7f67c3fff640 1 -- 172.21.15.6:0/1624931945 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67cc0a9b70 con 0x7f67cc0b4ff0 2024-03-25T19:20:19.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.840+0000 7f67c3fff640 10 monclient: _finish_auth 0 2024-03-25T19:20:19.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.840+0000 7f67c3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:20:19.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.840+0000 7f67c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:19.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.840+0000 7f67c3fff640 1 -- 172.21.15.6:0/1624931945 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f67c8001690 con 0x7f67cc0b4ff0 2024-03-25T19:20:19.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.840+0000 7f67d3915640 5 monclient: authenticate success, global_id 4507 2024-03-25T19:20:19.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.840+0000 7f67d3915640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:19.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.840+0000 7f67c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:19.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.840+0000 7f67c3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:20:19.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.840+0000 7f67c3fff640 10 monclient: dump: 2024-03-25T19:20:19.842 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:20:19.842 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:19.842 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:19.842 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:19.842 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:19.842 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:19.842 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:20:19.842 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:20:19.842 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:20:19.842 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:19.842 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.840+0000 7f67d3915640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:19.846 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.846+0000 7f67c3fff640 1 -- 172.21.15.6:0/1624931945 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67c8001870 con 0x7f67cc0b4ff0 2024-03-25T19:20:19.846 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.846+0000 7f67c3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:19.846 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.846+0000 7f67c3fff640 1 -- 172.21.15.6:0/1624931945 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f67c8001c80 con 0x7f67cc0b4ff0 2024-03-25T19:20:19.846 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.846+0000 7f67d3915640 10 monclient: get_monmap_and_config success 2024-03-25T19:20:19.846 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.846+0000 7f67d3915640 4 set_mon_vals no callback set 2024-03-25T19:20:19.846 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.846+0000 7f67c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:19.846 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.846+0000 7f67c3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:20:19.846 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.846+0000 7f67c3fff640 10 monclient: dump: 2024-03-25T19:20:19.846 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:20:19.846 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:19.847 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:19.847 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:19.847 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:19.847 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:19.847 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:20:19.847 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:20:19.847 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:20:19.847 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:19.848 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.846+0000 7f67d3915640 10 monclient: shutdown 2024-03-25T19:20:19.848 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.846+0000 7f67d3915640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:19.849 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.846+0000 7f67d3915640 1 -- 172.21.15.6:0/1624931945 >> v1:172.21.15.192:6789/0 conn(0x7f67cc0b4ff0 legacy=0x7f67cc13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:19.849 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.847+0000 7f67d3915640 1 -- 172.21.15.6:0/1624931945 shutdown_connections 2024-03-25T19:20:19.849 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.847+0000 7f67d3915640 1 -- 172.21.15.6:0/1624931945 >> 172.21.15.6:0/1624931945 conn(0x7f67cc0617a0 msgr2=0x7f67cc061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:19.849 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.847+0000 7f67d3915640 1 -- 172.21.15.6:0/1624931945 shutdown_connections 2024-03-25T19:20:19.849 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.847+0000 7f67d3915640 1 -- 172.21.15.6:0/1624931945 wait complete. 2024-03-25T19:20:19.849 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.848+0000 7f67d3915640 10 monclient: build_initial_monmap 2024-03-25T19:20:19.849 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.848+0000 7f67d3915640 10 monclient: monmap: 2024-03-25T19:20:19.849 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:20:19.849 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:20:19.849 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:20:19.849 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:20:19.849 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:19.849 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:20:19.850 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:20:19.850 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:20:19.850 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:20:19.850 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:19.850 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.848+0000 7f67d3915640 1 librados: starting msgr at 2024-03-25T19:20:19.850 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.848+0000 7f67d3915640 1 librados: starting objecter 2024-03-25T19:20:19.850 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.848+0000 7f67d3915640 1 Processor -- start 2024-03-25T19:20:19.850 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.848+0000 7f67d3915640 1 -- start start 2024-03-25T19:20:19.850 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.848+0000 7f67d3915640 1 librados: setting wanted keys 2024-03-25T19:20:19.850 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.848+0000 7f67d3915640 1 librados: calling monclient init 2024-03-25T19:20:19.850 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.848+0000 7f67d3915640 10 monclient: init 2024-03-25T19:20:19.851 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.849+0000 7f67d3915640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:19.851 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.849+0000 7f67d3915640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:20:19.851 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.849+0000 7f67d3915640 10 monclient(hunting): picked mon.noname-b con 0x7f67cc0b4670 addr v1:172.21.15.192:6789/0 2024-03-25T19:20:19.851 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.849+0000 7f67d3915640 10 monclient(hunting): picked mon.noname-a con 0x7f67cc0b4ff0 addr v1:172.21.15.6:6789/0 2024-03-25T19:20:19.851 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.849+0000 7f67d3915640 10 monclient(hunting): picked mon.noname-c con 0x7f67cc13c480 addr v1:172.21.15.6:6790/0 2024-03-25T19:20:19.852 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.849+0000 7f67d3915640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67cc0b01c0 con 0x7f67cc0b4ff0 2024-03-25T19:20:19.852 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.849+0000 7f67d3915640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67cc0b0470 con 0x7f67cc0b4670 2024-03-25T19:20:19.852 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.849+0000 7f67d3915640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67cc0b0750 con 0x7f67cc13c480 2024-03-25T19:20:19.852 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.849+0000 7f67d3915640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:19.852 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.849+0000 7f67d3915640 10 monclient(hunting): authenticate will time out at 6324.089505s 2024-03-25T19:20:19.854 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.854+0000 7f67d1eb2640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f67cc13c480 0x7f67cc1e2380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:58400/0 (socket says 172.21.15.6:58400) 2024-03-25T19:20:19.855 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.855+0000 7f67d1eb2640 1 -- 172.21.15.6:0/2819850209 learned_addr learned my addr 172.21.15.6:0/2819850209 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:20:19.857 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.857+0000 7f67c1ffb640 10 client.?.objecter ms_handle_connect 0x7f67cc13c480 2024-03-25T19:20:19.857 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.857+0000 7f67c1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:19.860 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.860+0000 7f67c1ffb640 10 client.?.objecter ms_handle_connect 0x7f67cc0b4670 2024-03-25T19:20:19.860 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.860+0000 7f67c1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:19.865 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.865+0000 7f67c1ffb640 10 client.?.objecter ms_handle_connect 0x7f67cc0b4ff0 2024-03-25T19:20:19.865 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.865+0000 7f67c1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:19.868 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.868+0000 7f67c1ffb640 1 -- 172.21.15.6:0/2819850209 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 123792262 0 0) 0x7f67cc0b0750 con 0x7f67cc13c480 2024-03-25T19:20:19.869 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.868+0000 7f67c1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:20:19.869 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.868+0000 7f67c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:19.869 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.868+0000 7f67c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:19.869 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.868+0000 7f67c1ffb640 10 monclient(hunting): my global_id is 4505 2024-03-25T19:20:19.869 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.868+0000 7f67c1ffb640 1 -- 172.21.15.6:0/2819850209 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67b0004eb0 con 0x7f67cc13c480 2024-03-25T19:20:19.869 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.868+0000 7f67c1ffb640 1 -- 172.21.15.6:0/2819850209 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2967635501 0 0) 0x7f67cc0b0470 con 0x7f67cc0b4670 2024-03-25T19:20:19.869 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.868+0000 7f67c1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:20:19.869 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.868+0000 7f67c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:19.869 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.868+0000 7f67c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:19.869 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.868+0000 7f67c1ffb640 10 monclient(hunting): my global_id is 4510 2024-03-25T19:20:19.869 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.868+0000 7f67c1ffb640 1 -- 172.21.15.6:0/2819850209 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67cc0b0750 con 0x7f67cc0b4670 2024-03-25T19:20:19.873 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.873+0000 7f67c1ffb640 1 -- 172.21.15.6:0/2819850209 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 236102949 0 0) 0x7f67cc0b01c0 con 0x7f67cc0b4ff0 2024-03-25T19:20:19.873 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.873+0000 7f67c1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:20:19.874 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.873+0000 7f67c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:19.874 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.873+0000 7f67c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:19.874 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.873+0000 7f67c1ffb640 10 monclient(hunting): my global_id is 4506 2024-03-25T19:20:19.874 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.873+0000 7f67c1ffb640 1 -- 172.21.15.6:0/2819850209 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67cc0b0470 con 0x7f67cc0b4ff0 2024-03-25T19:20:19.880 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.880+0000 7f67c1ffb640 1 -- 172.21.15.6:0/2819850209 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1367241302 0 0) 0x7f67b0004eb0 con 0x7f67cc13c480 2024-03-25T19:20:19.880 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.880+0000 7f67c1ffb640 1 -- 172.21.15.6:0/2819850209 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67cc0b01c0 con 0x7f67cc13c480 2024-03-25T19:20:19.880 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.880+0000 7f67c1ffb640 1 -- 172.21.15.6:0/2819850209 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f67c4001c70 con 0x7f67cc13c480 2024-03-25T19:20:19.880 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.880+0000 7f67c1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:19.880 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.880+0000 7f67c1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:20:19.880 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.880+0000 7f67c1ffb640 10 monclient(hunting): dump: 2024-03-25T19:20:19.880 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:20:19.880 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:19.880 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:19.880 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:19.880 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:19.880 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:19.881 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:20:19.881 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:20:19.881 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:20:19.881 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:19.881 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.880+0000 7f67c1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:20:19.882 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.882+0000 7f67c1ffb640 1 -- 172.21.15.6:0/2819850209 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2903442640 0 0) 0x7f67cc0b0750 con 0x7f67cc0b4670 2024-03-25T19:20:19.883 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.882+0000 7f67c1ffb640 1 -- 172.21.15.6:0/2819850209 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67b0004eb0 con 0x7f67cc0b4670 2024-03-25T19:20:19.883 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.882+0000 7f67c1ffb640 1 -- 172.21.15.6:0/2819850209 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f67c8001b80 con 0x7f67cc0b4670 2024-03-25T19:20:19.883 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.882+0000 7f67c1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:19.883 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.882+0000 7f67c1ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:20:19.883 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.882+0000 7f67c1ffb640 10 monclient(hunting): dump: 2024-03-25T19:20:19.883 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:20:19.883 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:19.883 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:19.883 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:19.883 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:19.883 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:19.883 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:20:19.883 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:20:19.883 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:20:19.883 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:19.889 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.889+0000 7f67c1ffb640 1 -- 172.21.15.6:0/2819850209 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1609311355 0 0) 0x7f67cc0b0470 con 0x7f67cc0b4ff0 2024-03-25T19:20:19.889 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.889+0000 7f67c1ffb640 1 -- 172.21.15.6:0/2819850209 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67cc0b0750 con 0x7f67cc0b4ff0 2024-03-25T19:20:19.889 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.889+0000 7f67c1ffb640 1 -- 172.21.15.6:0/2819850209 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f67bc001870 con 0x7f67cc0b4ff0 2024-03-25T19:20:19.889 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.889+0000 7f67c1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:19.889 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.889+0000 7f67c1ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:20:19.889 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.889+0000 7f67c1ffb640 10 monclient(hunting): dump: 2024-03-25T19:20:19.889 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:20:19.890 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:19.890 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:19.890 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:19.890 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:19.890 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:19.890 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:20:19.890 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:20:19.890 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:20:19.890 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:19.891 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.890+0000 7f67c1ffb640 1 -- 172.21.15.6:0/2819850209 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3642633964 0 0) 0x7f67cc0b01c0 con 0x7f67cc13c480 2024-03-25T19:20:19.891 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.891+0000 7f67c1ffb640 1 -- 172.21.15.6:0/2819850209 >> v1:172.21.15.192:6789/0 conn(0x7f67cc0b4670 legacy=0x7f67cc0b0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:19.891 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.891+0000 7f67c1ffb640 1 -- 172.21.15.6:0/2819850209 >> v1:172.21.15.6:6789/0 conn(0x7f67cc0b4ff0 legacy=0x7f67cc0ae0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:19.891 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.891+0000 7f67c1ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:20:19.891 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.891+0000 7f67c1ffb640 1 monclient: found mon.c 2024-03-25T19:20:19.891 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.891+0000 7f67c1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:19.891 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.891+0000 7f67c1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:20:19.892 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.891+0000 7f67c1ffb640 1 -- 172.21.15.6:0/2819850209 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67cc0aee30 con 0x7f67cc13c480 2024-03-25T19:20:19.892 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.891+0000 7f67c1ffb640 10 monclient: _finish_auth 0 2024-03-25T19:20:19.892 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.891+0000 7f67c1ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:20:19.892 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.891+0000 7f67c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:19.892 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.891+0000 7f67d3915640 5 monclient: authenticate success, global_id 4505 2024-03-25T19:20:19.892 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.891+0000 7f67d3915640 10 monclient: _renew_subs 2024-03-25T19:20:19.892 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.891+0000 7f67d3915640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:20:19.892 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.891+0000 7f67d3915640 1 -- 172.21.15.6:0/2819850209 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67cc0b1280 con 0x7f67cc13c480 2024-03-25T19:20:19.892 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.891+0000 7f67d3915640 10 client.4505.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:20:19.892 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.891+0000 7f67d3915640 10 monclient: _renew_subs 2024-03-25T19:20:19.892 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.891+0000 7f67d3915640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:20:19.892 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.891+0000 7f67d3915640 1 -- 172.21.15.6:0/2819850209 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f67cc0ae9e0 con 0x7f67cc13c480 2024-03-25T19:20:19.892 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.891+0000 7f67d3915640 1 librados: init done 2024-03-25T19:20:19.893 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.892+0000 7f67d3915640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:19.893 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.892+0000 7f67d3915640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:19.893 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.892+0000 7f67d3915640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:20:19.893 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:19.892+0000 7f67d3915640 1 -- 172.21.15.6:0/2819850209 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67cc0af070 con 0x7f67cc13c480 2024-03-25T19:20:20.210 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:20.209+0000 7f67c1ffb640 1 -- 172.21.15.6:0/2819850209 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67c4003990 con 0x7f67cc13c480 2024-03-25T19:20:20.210 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:20.209+0000 7f67c1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:20.210 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:20.209+0000 7f67c1ffb640 1 -- 172.21.15.6:0/2819850209 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f67c4003c00 con 0x7f67cc13c480 2024-03-25T19:20:20.210 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:20.209+0000 7f67c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:20.210 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:20.209+0000 7f67c1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:20:20.210 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:20.209+0000 7f67c1ffb640 10 monclient: dump: 2024-03-25T19:20:20.210 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:20:20.210 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:20.210 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:20.210 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:20.210 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:20.210 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:20.211 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:20:20.211 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:20:20.211 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:20:20.211 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:20.211 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:20.209+0000 7f67c37fe640 4 set_mon_vals no callback set 2024-03-25T19:20:20.211 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:20.210+0000 7f67c1ffb640 1 -- 172.21.15.6:0/2819850209 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f67c4003e20 con 0x7f67cc13c480 2024-03-25T19:20:20.211 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:20.211+0000 7f67d16b1640 10 monclient: get_auth_request con 0x7f67b006e0f0 auth_method 0 2024-03-25T19:20:20.211 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:20.211+0000 7f67c1ffb640 10 client.4505.objecter ms_handle_connect 0x7f67b006e0f0 2024-03-25T19:20:20.217 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:20.217+0000 7f67c1ffb640 1 -- 172.21.15.6:0/2819850209 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f67c4054860 con 0x7f67cc13c480 2024-03-25T19:20:20.218 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:20.217+0000 7f67c1ffb640 10 client.4505.objecter ms_dispatch 0x7f67cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:20:20.218 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:20.217+0000 7f67c1ffb640 3 client.4505.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:20:20.218 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:20.217+0000 7f67c1ffb640 3 client.4505.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:20:20.218 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:20.217+0000 7f67c1ffb640 20 client.4505.objecter dump_active .. 0 homeless 2024-03-25T19:20:20.218 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:20.217+0000 7f67c1ffb640 1 -- 172.21.15.6:0/2819850209 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f67c4050370 con 0x7f67cc13c480 2024-03-25T19:20:20.218 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:20.217+0000 7f67c1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:20.218 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:20.217+0000 7f67c1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:20:20.437 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:20.437+0000 7f67d3915640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:20.437 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:20.437+0000 7f67d3915640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:20.437 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:20.437+0000 7f67d3915640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:20:20.437 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:20.437+0000 7f67d3915640 1 -- 172.21.15.6:0/2819850209 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f67cc0688c0 con 0x7f67cc13c480 2024-03-25T19:20:20.473 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:20.473+0000 7f67c1ffb640 1 -- 172.21.15.6:0/2819850209 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f67c40522a0 con 0x7f67cc13c480 2024-03-25T19:20:20.473 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:20.473+0000 7f67c1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:20.473 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:20.473+0000 7f67c1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:20:20.474 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:20:20.474 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:20:20.476 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:20.475+0000 7f67d3915640 10 librados: watch_flush enter 2024-03-25T19:20:20.476 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:20.475+0000 7f67d3915640 10 librados: watch_flush exit 2024-03-25T19:20:20.476 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:20.475+0000 7f67d3915640 20 client.4505.objecter shutdown clearing up homeless session... 2024-03-25T19:20:20.476 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:20.475+0000 7f67d3915640 10 client.4505.objecter successfully canceled tick 2024-03-25T19:20:20.476 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:20.475+0000 7f67d3915640 1 -- 172.21.15.6:0/2819850209 >> v1:172.21.15.6:6800/2936652650 conn(0x7f67b006e0f0 legacy=0x7f67b00705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:20.476 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:20.475+0000 7f67d3915640 10 monclient: shutdown 2024-03-25T19:20:20.476 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:20.475+0000 7f67d3915640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:20.476 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:20.475+0000 7f67d3915640 1 -- 172.21.15.6:0/2819850209 >> v1:172.21.15.6:6790/0 conn(0x7f67cc13c480 legacy=0x7f67cc1e2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:20.476 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:20.475+0000 7f67d3915640 1 -- 172.21.15.6:0/2819850209 shutdown_connections 2024-03-25T19:20:20.477 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:20.476+0000 7f67d3915640 1 -- 172.21.15.6:0/2819850209 >> 172.21.15.6:0/2819850209 conn(0x7f67cc0617a0 msgr2=0x7f67cc0b2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:20.477 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:20.476+0000 7f67d3915640 1 -- 172.21.15.6:0/2819850209 shutdown_connections 2024-03-25T19:20:20.477 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:20.476+0000 7f67d3915640 1 -- 172.21.15.6:0/2819850209 wait complete. 2024-03-25T19:20:20.477 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:20.476+0000 7f67d3915640 1 librados: shutdown 2024-03-25T19:20:20.486 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:20:27.292 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:20:27.411 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.411+0000 7fcba6eb3640 10 monclient: get_monmap_and_config 2024-03-25T19:20:27.411 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.411+0000 7fcba6eb3640 10 monclient: build_initial_monmap 2024-03-25T19:20:27.411 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.411+0000 7fcba6eb3640 10 monclient: monmap: 2024-03-25T19:20:27.411 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:20:27.411 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:27.411 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:20:27.412246+0000 2024-03-25T19:20:27.411 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:20:27.412246+0000 2024-03-25T19:20:27.411 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:27.411 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:27.411 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:20:27.411 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:20:27.411 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:20:27.411 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:27.412 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.412+0000 7fcba6eb3640 1 Processor -- start 2024-03-25T19:20:27.412 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.412+0000 7fcba6eb3640 1 -- start start 2024-03-25T19:20:27.412 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.412+0000 7fcba6eb3640 10 monclient: init 2024-03-25T19:20:27.413 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.412+0000 7fcba6eb3640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:27.413 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.412+0000 7fcba6eb3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:20:27.413 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.412+0000 7fcba6eb3640 10 monclient(hunting): picked mon.noname-b con 0x7fcba00a98d0 addr v1:172.21.15.6:6790/0 2024-03-25T19:20:27.413 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.412+0000 7fcba6eb3640 10 monclient(hunting): picked mon.noname-a con 0x7fcba00b2a60 addr v1:172.21.15.6:6789/0 2024-03-25T19:20:27.413 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.412+0000 7fcba6eb3640 10 monclient(hunting): picked mon.noname-c con 0x7fcba014bc00 addr v1:172.21.15.192:6789/0 2024-03-25T19:20:27.413 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.412+0000 7fcba6eb3640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcba00b2e80 con 0x7fcba00b2a60 2024-03-25T19:20:27.413 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.413+0000 7fcba6eb3640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcba01431c0 con 0x7fcba014bc00 2024-03-25T19:20:27.413 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.413+0000 7fcba6eb3640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcba01422f0 con 0x7fcba00a98d0 2024-03-25T19:20:27.413 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.413+0000 7fcba6eb3640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:27.413 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.413+0000 7fcba6eb3640 10 monclient(hunting): authenticate will time out at 6331.652806s 2024-03-25T19:20:27.426 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.426+0000 7fcba5450640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fcba014bc00 0x7fcba014dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:58810/0 (socket says 172.21.15.6:58810) 2024-03-25T19:20:27.426 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.426+0000 7fcba5450640 1 -- 172.21.15.6:0/979808268 learned_addr learned my addr 172.21.15.6:0/979808268 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:20:27.448 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.448+0000 7fcb9f7fe640 1 -- 172.21.15.6:0/979808268 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 384227920 0 0) 0x7fcba01422f0 con 0x7fcba00a98d0 2024-03-25T19:20:27.448 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.448+0000 7fcb9f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:20:27.448 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.448+0000 7fcb9f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:27.448 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.448+0000 7fcb9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:27.448 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.448+0000 7fcb9f7fe640 10 monclient(hunting): my global_id is 4508 2024-03-25T19:20:27.448 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.448+0000 7fcb9f7fe640 1 -- 172.21.15.6:0/979808268 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb90003520 con 0x7fcba00a98d0 2024-03-25T19:20:27.455 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.455+0000 7fcb9f7fe640 1 -- 172.21.15.6:0/979808268 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1594614817 0 0) 0x7fcba01431c0 con 0x7fcba014bc00 2024-03-25T19:20:27.455 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.455+0000 7fcb9f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:20:27.455 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.455+0000 7fcb9f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:27.455 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.455+0000 7fcb9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:27.455 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.455+0000 7fcb9f7fe640 10 monclient(hunting): my global_id is 4513 2024-03-25T19:20:27.455 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.455+0000 7fcb9f7fe640 1 -- 172.21.15.6:0/979808268 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcba01422f0 con 0x7fcba014bc00 2024-03-25T19:20:27.457 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.457+0000 7fcb9f7fe640 1 -- 172.21.15.6:0/979808268 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1000561352 0 0) 0x7fcba00b2e80 con 0x7fcba00b2a60 2024-03-25T19:20:27.457 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.457+0000 7fcb9f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:20:27.457 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.457+0000 7fcb9f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:27.457 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.457+0000 7fcb9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:27.457 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.457+0000 7fcb9f7fe640 10 monclient(hunting): my global_id is 4509 2024-03-25T19:20:27.457 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.457+0000 7fcb9f7fe640 1 -- 172.21.15.6:0/979808268 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcba01431c0 con 0x7fcba00b2a60 2024-03-25T19:20:27.469 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.469+0000 7fcb9f7fe640 1 -- 172.21.15.6:0/979808268 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3671521680 0 0) 0x7fcb90003520 con 0x7fcba00a98d0 2024-03-25T19:20:27.469 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.469+0000 7fcb9f7fe640 1 -- 172.21.15.6:0/979808268 >> v1:172.21.15.192:6789/0 conn(0x7fcba014bc00 legacy=0x7fcba014dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:27.469 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.469+0000 7fcb9f7fe640 1 -- 172.21.15.6:0/979808268 >> v1:172.21.15.6:6789/0 conn(0x7fcba00b2a60 legacy=0x7fcba013f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:27.470 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.469+0000 7fcb9f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:20:27.470 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.469+0000 7fcb9f7fe640 1 monclient: found mon.noname-b 2024-03-25T19:20:27.470 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.469+0000 7fcb9f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:27.470 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.469+0000 7fcb9f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-03-25T19:20:27.470 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.469+0000 7fcb9f7fe640 1 -- 172.21.15.6:0/979808268 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcba0142590 con 0x7fcba00a98d0 2024-03-25T19:20:27.470 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.469+0000 7fcb9f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:20:27.470 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.469+0000 7fcb9f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:20:27.470 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.469+0000 7fcb9f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:27.470 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.469+0000 7fcb9f7fe640 1 -- 172.21.15.6:0/979808268 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fcb94001690 con 0x7fcba00a98d0 2024-03-25T19:20:27.470 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.469+0000 7fcba6eb3640 5 monclient: authenticate success, global_id 4508 2024-03-25T19:20:27.470 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.469+0000 7fcba6eb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:27.470 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.469+0000 7fcb9f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:27.470 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.469+0000 7fcb9f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:20:27.470 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.469+0000 7fcb9f7fe640 10 monclient: dump: 2024-03-25T19:20:27.470 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:20:27.471 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:27.471 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:27.471 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:27.471 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:27.471 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:27.471 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:20:27.471 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:20:27.471 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:20:27.471 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:27.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.469+0000 7fcba6eb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:27.490 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.490+0000 7fcb9f7fe640 1 -- 172.21.15.6:0/979808268 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb940017f0 con 0x7fcba00a98d0 2024-03-25T19:20:27.490 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.490+0000 7fcb9f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:27.490 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.490+0000 7fcb9f7fe640 1 -- 172.21.15.6:0/979808268 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fcb94001c00 con 0x7fcba00a98d0 2024-03-25T19:20:27.490 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.490+0000 7fcba6eb3640 10 monclient: get_monmap_and_config success 2024-03-25T19:20:27.491 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.490+0000 7fcba6eb3640 4 set_mon_vals no callback set 2024-03-25T19:20:27.491 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.490+0000 7fcb9f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:27.491 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.490+0000 7fcb9f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:20:27.491 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.490+0000 7fcb9f7fe640 10 monclient: dump: 2024-03-25T19:20:27.491 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:20:27.491 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:27.491 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:27.491 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:27.491 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:27.491 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:27.491 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:20:27.491 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:20:27.491 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:20:27.491 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:27.492 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.491+0000 7fcba6eb3640 10 monclient: shutdown 2024-03-25T19:20:27.492 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.491+0000 7fcba6eb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:27.493 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.491+0000 7fcba6eb3640 1 -- 172.21.15.6:0/979808268 >> v1:172.21.15.6:6790/0 conn(0x7fcba00a98d0 legacy=0x7fcba0142ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:27.493 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.491+0000 7fcba6eb3640 1 -- 172.21.15.6:0/979808268 shutdown_connections 2024-03-25T19:20:27.493 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.491+0000 7fcba6eb3640 1 -- 172.21.15.6:0/979808268 >> 172.21.15.6:0/979808268 conn(0x7fcba00b4b60 msgr2=0x7fcba00b4f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:27.493 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.491+0000 7fcba6eb3640 1 -- 172.21.15.6:0/979808268 shutdown_connections 2024-03-25T19:20:27.493 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.491+0000 7fcba6eb3640 1 -- 172.21.15.6:0/979808268 wait complete. 2024-03-25T19:20:27.493 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.492+0000 7fcba6eb3640 10 monclient: build_initial_monmap 2024-03-25T19:20:27.493 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.492+0000 7fcba6eb3640 10 monclient: monmap: 2024-03-25T19:20:27.493 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:20:27.493 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:20:27.493 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:20:27.493 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:20:27.493 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:27.493 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:20:27.493 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:20:27.493 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:20:27.493 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:20:27.494 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:27.494 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.493+0000 7fcba6eb3640 1 librados: starting msgr at 2024-03-25T19:20:27.494 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.493+0000 7fcba6eb3640 1 librados: starting objecter 2024-03-25T19:20:27.494 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.493+0000 7fcba6eb3640 1 Processor -- start 2024-03-25T19:20:27.494 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.493+0000 7fcba6eb3640 1 -- start start 2024-03-25T19:20:27.494 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.493+0000 7fcba6eb3640 1 librados: setting wanted keys 2024-03-25T19:20:27.495 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.493+0000 7fcba6eb3640 1 librados: calling monclient init 2024-03-25T19:20:27.495 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.493+0000 7fcba6eb3640 10 monclient: init 2024-03-25T19:20:27.495 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.494+0000 7fcba6eb3640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:27.495 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.494+0000 7fcba6eb3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:20:27.495 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.494+0000 7fcba6eb3640 10 monclient(hunting): picked mon.noname-b con 0x7fcba00a98d0 addr v1:172.21.15.192:6789/0 2024-03-25T19:20:27.495 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.494+0000 7fcba6eb3640 10 monclient(hunting): picked mon.noname-c con 0x7fcba00b2a60 addr v1:172.21.15.6:6790/0 2024-03-25T19:20:27.495 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.494+0000 7fcba6eb3640 10 monclient(hunting): picked mon.noname-a con 0x7fcba014bc00 addr v1:172.21.15.6:6789/0 2024-03-25T19:20:27.495 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.494+0000 7fcba6eb3640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcba01475a0 con 0x7fcba014bc00 2024-03-25T19:20:27.495 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.494+0000 7fcba6eb3640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcba0147f30 con 0x7fcba00a98d0 2024-03-25T19:20:27.495 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.494+0000 7fcba6eb3640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcba01481a0 con 0x7fcba00b2a60 2024-03-25T19:20:27.495 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.494+0000 7fcba6eb3640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:27.495 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.494+0000 7fcba6eb3640 10 monclient(hunting): authenticate will time out at 6331.734086s 2024-03-25T19:20:27.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.506+0000 7fcba5450640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fcba014bc00 0x7fcba01d7e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39836/0 (socket says 172.21.15.6:39836) 2024-03-25T19:20:27.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.506+0000 7fcba5450640 1 -- 172.21.15.6:0/591680530 learned_addr learned my addr 172.21.15.6:0/591680530 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:20:27.507 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.507+0000 7fcb9d7fa640 10 client.?.objecter ms_handle_connect 0x7fcba014bc00 2024-03-25T19:20:27.507 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.507+0000 7fcb9d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:27.509 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.509+0000 7fcb9d7fa640 10 client.?.objecter ms_handle_connect 0x7fcba00a98d0 2024-03-25T19:20:27.509 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.509+0000 7fcb9d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:27.516 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.516+0000 7fcb9d7fa640 10 client.?.objecter ms_handle_connect 0x7fcba00b2a60 2024-03-25T19:20:27.516 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.516+0000 7fcb9d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:27.518 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.518+0000 7fcb9d7fa640 1 -- 172.21.15.6:0/591680530 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2756734320 0 0) 0x7fcba0147f30 con 0x7fcba00a98d0 2024-03-25T19:20:27.518 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.518+0000 7fcb9d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:20:27.518 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.518+0000 7fcb9d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:27.518 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.518+0000 7fcb9d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:27.518 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.518+0000 7fcb9d7fa640 10 monclient(hunting): my global_id is 4516 2024-03-25T19:20:27.518 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.518+0000 7fcb9d7fa640 1 -- 172.21.15.6:0/591680530 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb84004eb0 con 0x7fcba00a98d0 2024-03-25T19:20:27.524 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.524+0000 7fcb9d7fa640 1 -- 172.21.15.6:0/591680530 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1959607510 0 0) 0x7fcba01475a0 con 0x7fcba014bc00 2024-03-25T19:20:27.524 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.524+0000 7fcb9d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:20:27.525 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.524+0000 7fcb9d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:27.525 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.524+0000 7fcb9d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:27.525 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.524+0000 7fcb9d7fa640 10 monclient(hunting): my global_id is 4512 2024-03-25T19:20:27.525 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.524+0000 7fcb9d7fa640 1 -- 172.21.15.6:0/591680530 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcba0147f30 con 0x7fcba014bc00 2024-03-25T19:20:27.527 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.527+0000 7fcb9d7fa640 1 -- 172.21.15.6:0/591680530 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3609253326 0 0) 0x7fcba01481a0 con 0x7fcba00b2a60 2024-03-25T19:20:27.527 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.527+0000 7fcb9d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:20:27.527 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.527+0000 7fcb9d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:27.527 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.527+0000 7fcb9d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:27.528 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.527+0000 7fcb9d7fa640 10 monclient(hunting): my global_id is 4511 2024-03-25T19:20:27.528 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.527+0000 7fcb9d7fa640 1 -- 172.21.15.6:0/591680530 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcba01475a0 con 0x7fcba00b2a60 2024-03-25T19:20:27.543 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.543+0000 7fcb9d7fa640 1 -- 172.21.15.6:0/591680530 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 922459603 0 0) 0x7fcb84004eb0 con 0x7fcba00a98d0 2024-03-25T19:20:27.544 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.543+0000 7fcb9d7fa640 1 -- 172.21.15.6:0/591680530 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcba01481a0 con 0x7fcba00a98d0 2024-03-25T19:20:27.544 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.543+0000 7fcb9d7fa640 1 -- 172.21.15.6:0/591680530 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fcb94001b00 con 0x7fcba00a98d0 2024-03-25T19:20:27.544 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.543+0000 7fcb9d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:27.544 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.543+0000 7fcb9d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:20:27.544 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.543+0000 7fcb9d7fa640 10 monclient(hunting): dump: 2024-03-25T19:20:27.544 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:20:27.544 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:27.544 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:27.544 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:27.544 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:27.544 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:27.544 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:20:27.544 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:20:27.544 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:20:27.545 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:27.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.543+0000 7fcb9d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:20:27.550 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.549+0000 7fcb9d7fa640 1 -- 172.21.15.6:0/591680530 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1815820127 0 0) 0x7fcba01481a0 con 0x7fcba00a98d0 2024-03-25T19:20:27.550 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.549+0000 7fcb9d7fa640 1 -- 172.21.15.6:0/591680530 >> v1:172.21.15.6:6790/0 conn(0x7fcba00b2a60 legacy=0x7fcba01381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:27.550 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.549+0000 7fcb9d7fa640 1 -- 172.21.15.6:0/591680530 >> v1:172.21.15.6:6789/0 conn(0x7fcba014bc00 legacy=0x7fcba01d7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:27.550 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.549+0000 7fcb9d7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:20:27.550 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.549+0000 7fcb9d7fa640 1 monclient: found mon.b 2024-03-25T19:20:27.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.549+0000 7fcb9d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:27.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.549+0000 7fcb9d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:20:27.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.549+0000 7fcb9d7fa640 1 -- 172.21.15.6:0/591680530 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcba0148480 con 0x7fcba00a98d0 2024-03-25T19:20:27.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.549+0000 7fcb9d7fa640 10 monclient: _finish_auth 0 2024-03-25T19:20:27.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.549+0000 7fcb9d7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:20:27.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.549+0000 7fcb9d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:27.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.549+0000 7fcba6eb3640 5 monclient: authenticate success, global_id 4516 2024-03-25T19:20:27.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.549+0000 7fcba6eb3640 10 monclient: _renew_subs 2024-03-25T19:20:27.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.549+0000 7fcba6eb3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:20:27.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.549+0000 7fcba6eb3640 1 -- 172.21.15.6:0/591680530 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcba0148760 con 0x7fcba00a98d0 2024-03-25T19:20:27.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.549+0000 7fcba6eb3640 10 client.4516.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:20:27.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.549+0000 7fcba6eb3640 10 monclient: _renew_subs 2024-03-25T19:20:27.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.549+0000 7fcba6eb3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:20:27.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.549+0000 7fcba6eb3640 1 -- 172.21.15.6:0/591680530 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcba0138a90 con 0x7fcba00a98d0 2024-03-25T19:20:27.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.549+0000 7fcba6eb3640 1 librados: init done 2024-03-25T19:20:27.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.550+0000 7fcba6eb3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:27.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.550+0000 7fcba6eb3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:27.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.550+0000 7fcba6eb3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:20:27.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.550+0000 7fcba6eb3640 1 -- 172.21.15.6:0/591680530 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcba00ab020 con 0x7fcba00a98d0 2024-03-25T19:20:27.553 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.553+0000 7fcb9d7fa640 1 -- 172.21.15.6:0/591680530 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb94001220 con 0x7fcba00a98d0 2024-03-25T19:20:27.553 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.553+0000 7fcb9d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:27.553 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.553+0000 7fcb9d7fa640 1 -- 172.21.15.6:0/591680530 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fcb94003a40 con 0x7fcba00a98d0 2024-03-25T19:20:27.553 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.553+0000 7fcb9d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:27.553 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.553+0000 7fcb9d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:20:27.553 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.553+0000 7fcb9d7fa640 10 monclient: dump: 2024-03-25T19:20:27.553 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:20:27.553 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:27.553 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:27.553 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:27.553 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:27.553 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:27.554 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:20:27.554 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:20:27.554 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:20:27.554 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:27.554 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.553+0000 7fcb9effd640 4 set_mon_vals no callback set 2024-03-25T19:20:27.819 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.819+0000 7fcb9d7fa640 1 -- 172.21.15.6:0/591680530 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7fcb94019a90 con 0x7fcba00a98d0 2024-03-25T19:20:27.820 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.820+0000 7fcb9ffff640 10 monclient: get_auth_request con 0x7fcb8406ddf0 auth_method 0 2024-03-25T19:20:27.820 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.820+0000 7fcb9d7fa640 10 client.4516.objecter ms_handle_connect 0x7fcb8406ddf0 2024-03-25T19:20:27.822 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.822+0000 7fcb9d7fa640 1 -- 172.21.15.6:0/591680530 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7fcb940536e0 con 0x7fcba00a98d0 2024-03-25T19:20:27.822 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.822+0000 7fcb9d7fa640 10 client.4516.objecter ms_dispatch 0x7fcba0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:20:27.822 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.822+0000 7fcb9d7fa640 3 client.4516.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:20:27.823 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.822+0000 7fcb9d7fa640 3 client.4516.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:20:27.823 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.822+0000 7fcb9d7fa640 20 client.4516.objecter dump_active .. 0 homeless 2024-03-25T19:20:27.826 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.826+0000 7fcb9d7fa640 1 -- 172.21.15.6:0/591680530 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcb9404f020 con 0x7fcba00a98d0 2024-03-25T19:20:27.826 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.826+0000 7fcb9d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:27.826 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:27.826+0000 7fcb9d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:20:28.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:28.059+0000 7fcba6eb3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:28.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:28.059+0000 7fcba6eb3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:28.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:28.059+0000 7fcba6eb3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:20:28.060 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:28.059+0000 7fcba6eb3640 1 -- 172.21.15.6:0/591680530 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcba00aa5c0 con 0x7fcba00a98d0 2024-03-25T19:20:28.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:28.104+0000 7fcb9d7fa640 1 -- 172.21.15.6:0/591680530 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7fcb94051120 con 0x7fcba00a98d0 2024-03-25T19:20:28.105 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:28.104+0000 7fcb9d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:28.105 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:28.104+0000 7fcb9d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:20:28.105 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:20:28.105 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:20:28.107 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:28.107+0000 7fcba6eb3640 10 librados: watch_flush enter 2024-03-25T19:20:28.108 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:28.107+0000 7fcba6eb3640 10 librados: watch_flush exit 2024-03-25T19:20:28.108 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:28.107+0000 7fcba6eb3640 20 client.4516.objecter shutdown clearing up homeless session... 2024-03-25T19:20:28.108 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:28.107+0000 7fcba6eb3640 10 client.4516.objecter successfully canceled tick 2024-03-25T19:20:28.108 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:28.107+0000 7fcba6eb3640 1 -- 172.21.15.6:0/591680530 >> v1:172.21.15.6:6800/2936652650 conn(0x7fcb8406ddf0 legacy=0x7fcb840702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:28.108 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:28.107+0000 7fcba6eb3640 10 monclient: shutdown 2024-03-25T19:20:28.108 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:28.107+0000 7fcba6eb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:28.108 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:28.107+0000 7fcba6eb3640 1 -- 172.21.15.6:0/591680530 >> v1:172.21.15.192:6789/0 conn(0x7fcba00a98d0 legacy=0x7fcba0146c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:28.108 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:28.107+0000 7fcba6eb3640 1 -- 172.21.15.6:0/591680530 shutdown_connections 2024-03-25T19:20:28.108 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:28.107+0000 7fcba6eb3640 1 -- 172.21.15.6:0/591680530 >> 172.21.15.6:0/591680530 conn(0x7fcba00b4b60 msgr2=0x7fcba00b78c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:28.108 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:28.107+0000 7fcba6eb3640 1 -- 172.21.15.6:0/591680530 shutdown_connections 2024-03-25T19:20:28.108 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:28.107+0000 7fcba6eb3640 1 -- 172.21.15.6:0/591680530 wait complete. 2024-03-25T19:20:28.108 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:28.107+0000 7fcba6eb3640 1 librados: shutdown 2024-03-25T19:20:28.120 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:20:28.645 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:06:45.992 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:28.649 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:06:45.996 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:28.693 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:06:46.040 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:28.698 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:06:46.045 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:28.701 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:06:46.019 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:28.705 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:06:46.023 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:28.717 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:06:46.063 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:28.741 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:06:46.059 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:28.746 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:06:46.093 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:34.926 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:20:35.042 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.042+0000 7f49b62f1640 10 monclient: get_monmap_and_config 2024-03-25T19:20:35.042 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.042+0000 7f49b62f1640 10 monclient: build_initial_monmap 2024-03-25T19:20:35.042 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.042+0000 7f49b62f1640 10 monclient: monmap: 2024-03-25T19:20:35.042 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:20:35.042 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:35.042 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:20:35.043230+0000 2024-03-25T19:20:35.042 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:20:35.043230+0000 2024-03-25T19:20:35.043 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:35.043 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:35.043 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:20:35.043 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:20:35.043 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:20:35.043 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:35.044 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.043+0000 7f49b62f1640 1 Processor -- start 2024-03-25T19:20:35.044 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.043+0000 7f49b62f1640 1 -- start start 2024-03-25T19:20:35.044 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.043+0000 7f49b62f1640 10 monclient: init 2024-03-25T19:20:35.044 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.043+0000 7f49b62f1640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:35.044 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.043+0000 7f49b62f1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:20:35.044 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.043+0000 7f49b62f1640 10 monclient(hunting): picked mon.noname-b con 0x7f49b00b4ff0 addr v1:172.21.15.6:6790/0 2024-03-25T19:20:35.044 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.043+0000 7f49b62f1640 10 monclient(hunting): picked mon.noname-c con 0x7f49b00b4670 addr v1:172.21.15.192:6789/0 2024-03-25T19:20:35.044 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.043+0000 7f49b62f1640 10 monclient(hunting): picked mon.noname-a con 0x7f49b013c480 addr v1:172.21.15.6:6789/0 2024-03-25T19:20:35.044 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.043+0000 7f49b62f1640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49b00ab0e0 con 0x7f49b013c480 2024-03-25T19:20:35.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.043+0000 7f49b62f1640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49b00ab3c0 con 0x7f49b00b4670 2024-03-25T19:20:35.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.043+0000 7f49b62f1640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49b00a9f30 con 0x7f49b00b4ff0 2024-03-25T19:20:35.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.043+0000 7f49b62f1640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:35.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.043+0000 7f49b62f1640 10 monclient(hunting): authenticate will time out at 6339.283601s 2024-03-25T19:20:35.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.051+0000 7f49af7fe640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f49b00b4670 0x7f49b01425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:34412/0 (socket says 172.21.15.6:34412) 2024-03-25T19:20:35.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.051+0000 7f49af7fe640 1 -- 172.21.15.6:0/1924426830 learned_addr learned my addr 172.21.15.6:0/1924426830 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:20:35.068 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.068+0000 7f49aeffd640 1 -- 172.21.15.6:0/1924426830 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1407027458 0 0) 0x7f49b00ab3c0 con 0x7f49b00b4670 2024-03-25T19:20:35.068 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.068+0000 7f49aeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:20:35.068 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.068+0000 7f49aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:35.068 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.068+0000 7f49aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:35.068 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.068+0000 7f49aeffd640 10 monclient(hunting): my global_id is 4519 2024-03-25T19:20:35.068 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.068+0000 7f49aeffd640 1 -- 172.21.15.6:0/1924426830 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4998003520 con 0x7f49b00b4670 2024-03-25T19:20:35.074 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.074+0000 7f49aeffd640 1 -- 172.21.15.6:0/1924426830 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2707486234 0 0) 0x7f49b00ab0e0 con 0x7f49b013c480 2024-03-25T19:20:35.074 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.074+0000 7f49aeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:20:35.074 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.074+0000 7f49aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:35.074 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.074+0000 7f49aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:35.075 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.074+0000 7f49aeffd640 10 monclient(hunting): my global_id is 4515 2024-03-25T19:20:35.075 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.074+0000 7f49aeffd640 1 -- 172.21.15.6:0/1924426830 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49b00ab3c0 con 0x7f49b013c480 2024-03-25T19:20:35.078 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.078+0000 7f49aeffd640 1 -- 172.21.15.6:0/1924426830 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3851864504 0 0) 0x7f49b00a9f30 con 0x7f49b00b4ff0 2024-03-25T19:20:35.078 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.078+0000 7f49aeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:20:35.078 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.078+0000 7f49aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:35.078 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.078+0000 7f49aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:35.079 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.078+0000 7f49aeffd640 10 monclient(hunting): my global_id is 4514 2024-03-25T19:20:35.079 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.078+0000 7f49aeffd640 1 -- 172.21.15.6:0/1924426830 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49b00ab0e0 con 0x7f49b00b4ff0 2024-03-25T19:20:35.088 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.087+0000 7f49aeffd640 1 -- 172.21.15.6:0/1924426830 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3806323056 0 0) 0x7f4998003520 con 0x7f49b00b4670 2024-03-25T19:20:35.088 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.088+0000 7f49aeffd640 1 -- 172.21.15.6:0/1924426830 >> v1:172.21.15.6:6790/0 conn(0x7f49b00b4ff0 legacy=0x7f49b013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:35.088 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.088+0000 7f49aeffd640 1 -- 172.21.15.6:0/1924426830 >> v1:172.21.15.6:6789/0 conn(0x7f49b013c480 legacy=0x7f49b014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:35.088 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.088+0000 7f49aeffd640 10 monclient: _finish_hunting 0 2024-03-25T19:20:35.088 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.088+0000 7f49aeffd640 1 monclient: found mon.noname-c 2024-03-25T19:20:35.088 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.088+0000 7f49aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:35.088 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.088+0000 7f49aeffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:20:35.088 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.088+0000 7f49aeffd640 1 -- 172.21.15.6:0/1924426830 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b00a9b70 con 0x7f49b00b4670 2024-03-25T19:20:35.088 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.088+0000 7f49aeffd640 10 monclient: _finish_auth 0 2024-03-25T19:20:35.089 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.088+0000 7f49aeffd640 10 monclient: _check_auth_tickets 2024-03-25T19:20:35.089 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.088+0000 7f49aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:35.089 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.088+0000 7f49aeffd640 1 -- 172.21.15.6:0/1924426830 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f49a4001690 con 0x7f49b00b4670 2024-03-25T19:20:35.089 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.088+0000 7f49aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:35.089 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.088+0000 7f49aeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:20:35.089 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.088+0000 7f49aeffd640 10 monclient: dump: 2024-03-25T19:20:35.089 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:20:35.089 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:35.089 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:35.089 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:35.089 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:35.089 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:35.089 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:20:35.089 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:20:35.089 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:20:35.090 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:35.090 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.088+0000 7f49b62f1640 5 monclient: authenticate success, global_id 4519 2024-03-25T19:20:35.090 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.088+0000 7f49b62f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:35.114 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.113+0000 7f49aeffd640 1 -- 172.21.15.6:0/1924426830 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49a40017f0 con 0x7f49b00b4670 2024-03-25T19:20:35.114 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.113+0000 7f49aeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:35.114 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.113+0000 7f49aeffd640 1 -- 172.21.15.6:0/1924426830 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f49a4001c00 con 0x7f49b00b4670 2024-03-25T19:20:35.114 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.113+0000 7f49aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:35.114 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.113+0000 7f49aeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:20:35.114 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.113+0000 7f49aeffd640 10 monclient: dump: 2024-03-25T19:20:35.114 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:20:35.114 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:35.114 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:35.114 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:35.114 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:35.114 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:35.114 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:20:35.114 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:20:35.115 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:20:35.115 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:35.115 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.113+0000 7f49b62f1640 10 monclient: get_monmap_and_config success 2024-03-25T19:20:35.115 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.113+0000 7f49b62f1640 4 set_mon_vals no callback set 2024-03-25T19:20:35.115 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.114+0000 7f49b62f1640 10 monclient: shutdown 2024-03-25T19:20:35.115 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.114+0000 7f49b62f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:35.115 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.114+0000 7f49b62f1640 1 -- 172.21.15.6:0/1924426830 >> v1:172.21.15.192:6789/0 conn(0x7f49b00b4670 legacy=0x7f49b01425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:35.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.115+0000 7f49b62f1640 1 -- 172.21.15.6:0/1924426830 shutdown_connections 2024-03-25T19:20:35.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.115+0000 7f49b62f1640 1 -- 172.21.15.6:0/1924426830 >> 172.21.15.6:0/1924426830 conn(0x7f49b00617a0 msgr2=0x7f49b0061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:35.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.115+0000 7f49b62f1640 1 -- 172.21.15.6:0/1924426830 shutdown_connections 2024-03-25T19:20:35.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.115+0000 7f49b62f1640 1 -- 172.21.15.6:0/1924426830 wait complete. 2024-03-25T19:20:35.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.116+0000 7f49b62f1640 10 monclient: build_initial_monmap 2024-03-25T19:20:35.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.116+0000 7f49b62f1640 10 monclient: monmap: 2024-03-25T19:20:35.117 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:20:35.117 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:20:35.117 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:20:35.117 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:20:35.117 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:35.117 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:20:35.117 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:20:35.117 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:20:35.117 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:20:35.117 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:35.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.116+0000 7f49b62f1640 1 librados: starting msgr at 2024-03-25T19:20:35.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.116+0000 7f49b62f1640 1 librados: starting objecter 2024-03-25T19:20:35.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.116+0000 7f49b62f1640 1 Processor -- start 2024-03-25T19:20:35.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.117+0000 7f49b62f1640 1 -- start start 2024-03-25T19:20:35.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.117+0000 7f49b62f1640 1 librados: setting wanted keys 2024-03-25T19:20:35.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.117+0000 7f49b62f1640 1 librados: calling monclient init 2024-03-25T19:20:35.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.117+0000 7f49b62f1640 10 monclient: init 2024-03-25T19:20:35.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.117+0000 7f49b62f1640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:35.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.117+0000 7f49b62f1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:20:35.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.117+0000 7f49b62f1640 10 monclient(hunting): picked mon.noname-c con 0x7f49b00b4670 addr v1:172.21.15.6:6790/0 2024-03-25T19:20:35.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.117+0000 7f49b62f1640 10 monclient(hunting): picked mon.noname-a con 0x7f49b00b4ff0 addr v1:172.21.15.6:6789/0 2024-03-25T19:20:35.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.117+0000 7f49b62f1640 10 monclient(hunting): picked mon.noname-b con 0x7f49b013c480 addr v1:172.21.15.192:6789/0 2024-03-25T19:20:35.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.117+0000 7f49b62f1640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49b0144660 con 0x7f49b00b4ff0 2024-03-25T19:20:35.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.117+0000 7f49b62f1640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49b0144940 con 0x7f49b013c480 2024-03-25T19:20:35.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.117+0000 7f49b62f1640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49b0144c20 con 0x7f49b00b4670 2024-03-25T19:20:35.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.117+0000 7f49b62f1640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:35.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.117+0000 7f49b62f1640 10 monclient(hunting): authenticate will time out at 6339.357712s 2024-03-25T19:20:35.136 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.136+0000 7f49b488e640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f49b013c480 0x7f49b01303c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:34448/0 (socket says 172.21.15.6:34448) 2024-03-25T19:20:35.136 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.136+0000 7f49b488e640 1 -- 172.21.15.6:0/1264516980 learned_addr learned my addr 172.21.15.6:0/1264516980 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:20:35.138 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.138+0000 7f49acff9640 10 client.?.objecter ms_handle_connect 0x7f49b00b4ff0 2024-03-25T19:20:35.138 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.138+0000 7f49acff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:35.139 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.139+0000 7f49acff9640 10 client.?.objecter ms_handle_connect 0x7f49b013c480 2024-03-25T19:20:35.139 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.139+0000 7f49acff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:35.141 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.141+0000 7f49acff9640 10 client.?.objecter ms_handle_connect 0x7f49b00b4670 2024-03-25T19:20:35.141 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.141+0000 7f49acff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:35.153 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.152+0000 7f49acff9640 1 -- 172.21.15.6:0/1264516980 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2359058279 0 0) 0x7f49b0144940 con 0x7f49b013c480 2024-03-25T19:20:35.153 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.152+0000 7f49acff9640 10 monclient(hunting): _negotiate 2024-03-25T19:20:35.153 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.152+0000 7f49acff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:35.153 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.152+0000 7f49acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:35.153 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.152+0000 7f49acff9640 10 monclient(hunting): my global_id is 4522 2024-03-25T19:20:35.153 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.152+0000 7f49acff9640 1 -- 172.21.15.6:0/1264516980 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f499c004eb0 con 0x7f49b013c480 2024-03-25T19:20:35.153 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.152+0000 7f49acff9640 1 -- 172.21.15.6:0/1264516980 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 91203773 0 0) 0x7f49b0144660 con 0x7f49b00b4ff0 2024-03-25T19:20:35.153 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.152+0000 7f49acff9640 10 monclient(hunting): _negotiate 2024-03-25T19:20:35.153 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.152+0000 7f49acff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:35.153 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.152+0000 7f49acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:35.153 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.152+0000 7f49acff9640 10 monclient(hunting): my global_id is 4518 2024-03-25T19:20:35.153 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.152+0000 7f49acff9640 1 -- 172.21.15.6:0/1264516980 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49b0144940 con 0x7f49b00b4ff0 2024-03-25T19:20:35.155 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.155+0000 7f49acff9640 1 -- 172.21.15.6:0/1264516980 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 776226849 0 0) 0x7f49b0144c20 con 0x7f49b00b4670 2024-03-25T19:20:35.155 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.155+0000 7f49acff9640 10 monclient(hunting): _negotiate 2024-03-25T19:20:35.155 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.155+0000 7f49acff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:35.155 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.155+0000 7f49acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:35.155 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.155+0000 7f49acff9640 10 monclient(hunting): my global_id is 4517 2024-03-25T19:20:35.155 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.155+0000 7f49acff9640 1 -- 172.21.15.6:0/1264516980 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49b0144660 con 0x7f49b00b4670 2024-03-25T19:20:35.180 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.180+0000 7f49acff9640 1 -- 172.21.15.6:0/1264516980 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1375046686 0 0) 0x7f49b0144940 con 0x7f49b00b4ff0 2024-03-25T19:20:35.180 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.180+0000 7f49acff9640 1 -- 172.21.15.6:0/1264516980 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49b0144c20 con 0x7f49b00b4ff0 2024-03-25T19:20:35.180 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.180+0000 7f49acff9640 1 -- 172.21.15.6:0/1264516980 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f49a4001b00 con 0x7f49b00b4ff0 2024-03-25T19:20:35.180 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.180+0000 7f49acff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:35.180 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.180+0000 7f49acff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:20:35.180 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.180+0000 7f49acff9640 10 monclient(hunting): dump: 2024-03-25T19:20:35.180 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:20:35.180 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:35.180 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:35.180 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:35.181 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:35.181 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:35.181 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:20:35.181 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:20:35.181 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:20:35.181 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:35.181 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.180+0000 7f49acff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:20:35.187 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.187+0000 7f49acff9640 1 -- 172.21.15.6:0/1264516980 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3356896737 0 0) 0x7f49b0144660 con 0x7f49b00b4670 2024-03-25T19:20:35.187 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.187+0000 7f49acff9640 1 -- 172.21.15.6:0/1264516980 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49b0144940 con 0x7f49b00b4670 2024-03-25T19:20:35.187 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.187+0000 7f49acff9640 1 -- 172.21.15.6:0/1264516980 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f49a0001830 con 0x7f49b00b4670 2024-03-25T19:20:35.187 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.187+0000 7f49acff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:35.187 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.187+0000 7f49acff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:20:35.187 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.187+0000 7f49acff9640 10 monclient(hunting): dump: 2024-03-25T19:20:35.187 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:20:35.188 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:35.188 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:35.188 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:35.188 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:35.188 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:35.188 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:20:35.188 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:20:35.188 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:20:35.188 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:35.190 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.190+0000 7f49acff9640 1 -- 172.21.15.6:0/1264516980 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4017700706 0 0) 0x7f499c004eb0 con 0x7f49b013c480 2024-03-25T19:20:35.190 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.190+0000 7f49acff9640 1 -- 172.21.15.6:0/1264516980 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49b0144660 con 0x7f49b013c480 2024-03-25T19:20:35.190 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.190+0000 7f49acff9640 1 -- 172.21.15.6:0/1264516980 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f49a8001c90 con 0x7f49b013c480 2024-03-25T19:20:35.190 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.190+0000 7f49acff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:35.190 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.190+0000 7f49acff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:20:35.191 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.190+0000 7f49acff9640 10 monclient(hunting): dump: 2024-03-25T19:20:35.191 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:20:35.191 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:35.191 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:35.191 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:35.191 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:35.191 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:35.191 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:20:35.191 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:20:35.191 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:20:35.191 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:35.212 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.211+0000 7f49acff9640 1 -- 172.21.15.6:0/1264516980 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1774151607 0 0) 0x7f49b0144940 con 0x7f49b00b4670 2024-03-25T19:20:35.212 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.212+0000 7f49acff9640 1 -- 172.21.15.6:0/1264516980 >> v1:172.21.15.192:6789/0 conn(0x7f49b013c480 legacy=0x7f49b01303c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:35.212 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.212+0000 7f49acff9640 1 -- 172.21.15.6:0/1264516980 >> v1:172.21.15.6:6789/0 conn(0x7f49b00b4ff0 legacy=0x7f49b012fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:35.212 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.212+0000 7f49acff9640 10 monclient: _finish_hunting 0 2024-03-25T19:20:35.212 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.212+0000 7f49acff9640 1 monclient: found mon.c 2024-03-25T19:20:35.212 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.212+0000 7f49acff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:35.212 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.212+0000 7f49acff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:20:35.212 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.212+0000 7f49acff9640 1 -- 172.21.15.6:0/1264516980 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b0143630 con 0x7f49b00b4670 2024-03-25T19:20:35.213 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.212+0000 7f49acff9640 10 monclient: _finish_auth 0 2024-03-25T19:20:35.213 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.212+0000 7f49acff9640 10 monclient: _check_auth_tickets 2024-03-25T19:20:35.213 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.212+0000 7f49acff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:35.213 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.212+0000 7f49b62f1640 5 monclient: authenticate success, global_id 4517 2024-03-25T19:20:35.213 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.212+0000 7f49b62f1640 10 monclient: _renew_subs 2024-03-25T19:20:35.213 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.212+0000 7f49b62f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:20:35.213 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.212+0000 7f49b62f1640 1 -- 172.21.15.6:0/1264516980 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49b0143910 con 0x7f49b00b4670 2024-03-25T19:20:35.213 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.212+0000 7f49b62f1640 10 client.4517.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:20:35.213 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.212+0000 7f49b62f1640 10 monclient: _renew_subs 2024-03-25T19:20:35.213 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.212+0000 7f49b62f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:20:35.213 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.212+0000 7f49b62f1640 1 -- 172.21.15.6:0/1264516980 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f49b0130ad0 con 0x7f49b00b4670 2024-03-25T19:20:35.213 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.212+0000 7f49b62f1640 1 librados: init done 2024-03-25T19:20:35.214 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.213+0000 7f49b62f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:35.214 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.213+0000 7f49b62f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:35.214 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.213+0000 7f49b62f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:20:35.214 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.213+0000 7f49b62f1640 1 -- 172.21.15.6:0/1264516980 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49b006dbd0 con 0x7f49b00b4670 2024-03-25T19:20:35.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.551+0000 7f49acff9640 1 -- 172.21.15.6:0/1264516980 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49a0001d40 con 0x7f49b00b4670 2024-03-25T19:20:35.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.551+0000 7f49acff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:35.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.551+0000 7f49acff9640 1 -- 172.21.15.6:0/1264516980 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f49a00037b0 con 0x7f49b00b4670 2024-03-25T19:20:35.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.551+0000 7f49acff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:35.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.551+0000 7f49acff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:20:35.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.551+0000 7f49acff9640 10 monclient: dump: 2024-03-25T19:20:35.552 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:20:35.552 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:35.552 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:35.552 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:35.552 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:35.552 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:35.552 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:20:35.552 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:20:35.553 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:20:35.553 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:35.553 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.551+0000 7f49ae7fc640 4 set_mon_vals no callback set 2024-03-25T19:20:35.553 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.552+0000 7f49acff9640 1 -- 172.21.15.6:0/1264516980 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f49a0019840 con 0x7f49b00b4670 2024-03-25T19:20:35.553 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.553+0000 7f49af7fe640 10 monclient: get_auth_request con 0x7f499c06e160 auth_method 0 2024-03-25T19:20:35.553 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.553+0000 7f49acff9640 10 client.4517.objecter ms_handle_connect 0x7f499c06e160 2024-03-25T19:20:35.563 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.562+0000 7f49acff9640 1 -- 172.21.15.6:0/1264516980 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f49a0053500 con 0x7f49b00b4670 2024-03-25T19:20:35.563 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.563+0000 7f49acff9640 10 client.4517.objecter ms_dispatch 0x7f49b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:20:35.563 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.563+0000 7f49acff9640 3 client.4517.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:20:35.563 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.563+0000 7f49acff9640 3 client.4517.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:20:35.563 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.563+0000 7f49acff9640 20 client.4517.objecter dump_active .. 0 homeless 2024-03-25T19:20:35.563 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.563+0000 7f49acff9640 1 -- 172.21.15.6:0/1264516980 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f49a004ee40 con 0x7f49b00b4670 2024-03-25T19:20:35.563 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.563+0000 7f49acff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:35.563 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.563+0000 7f49acff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:20:35.816 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.816+0000 7f49b62f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:35.816 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.816+0000 7f49b62f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:35.816 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.816+0000 7f49b62f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:20:35.816 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.816+0000 7f49b62f1640 1 -- 172.21.15.6:0/1264516980 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49b0143b50 con 0x7f49b00b4670 2024-03-25T19:20:35.845 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.845+0000 7f49acff9640 1 -- 172.21.15.6:0/1264516980 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f49b0143b50 con 0x7f49b00b4670 2024-03-25T19:20:35.845 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.845+0000 7f49acff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:35.845 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.845+0000 7f49acff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:20:35.845 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:20:35.846 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:20:35.848 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.847+0000 7f49b62f1640 10 librados: watch_flush enter 2024-03-25T19:20:35.848 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.847+0000 7f49b62f1640 10 librados: watch_flush exit 2024-03-25T19:20:35.848 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.847+0000 7f49b62f1640 20 client.4517.objecter shutdown clearing up homeless session... 2024-03-25T19:20:35.848 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.847+0000 7f49b62f1640 10 client.4517.objecter successfully canceled tick 2024-03-25T19:20:35.848 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.847+0000 7f49b62f1640 1 -- 172.21.15.6:0/1264516980 >> v1:172.21.15.6:6800/2936652650 conn(0x7f499c06e160 legacy=0x7f499c070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:35.848 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.847+0000 7f49b62f1640 10 monclient: shutdown 2024-03-25T19:20:35.848 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.847+0000 7f49b62f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:35.848 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.847+0000 7f49b62f1640 1 -- 172.21.15.6:0/1264516980 >> v1:172.21.15.6:6790/0 conn(0x7f49b00b4670 legacy=0x7f49b0145320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:35.848 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.847+0000 7f49b62f1640 1 -- 172.21.15.6:0/1264516980 shutdown_connections 2024-03-25T19:20:35.848 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.847+0000 7f49b62f1640 1 -- 172.21.15.6:0/1264516980 >> 172.21.15.6:0/1264516980 conn(0x7f49b00617a0 msgr2=0x7f49b0140760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:35.848 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.848+0000 7f49b62f1640 1 -- 172.21.15.6:0/1264516980 shutdown_connections 2024-03-25T19:20:35.848 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.848+0000 7f49b62f1640 1 -- 172.21.15.6:0/1264516980 wait complete. 2024-03-25T19:20:35.849 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:35.848+0000 7f49b62f1640 1 librados: shutdown 2024-03-25T19:20:35.860 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:20:42.666 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:20:42.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.783+0000 7fde270e4640 10 monclient: get_monmap_and_config 2024-03-25T19:20:42.784 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.783+0000 7fde270e4640 10 monclient: build_initial_monmap 2024-03-25T19:20:42.784 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.783+0000 7fde270e4640 10 monclient: monmap: 2024-03-25T19:20:42.784 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:20:42.784 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:42.784 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:20:42.784902+0000 2024-03-25T19:20:42.784 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:20:42.784902+0000 2024-03-25T19:20:42.784 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:42.784 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:42.784 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:20:42.784 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:20:42.784 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:20:42.784 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:42.785 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.784+0000 7fde270e4640 1 Processor -- start 2024-03-25T19:20:42.786 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.784+0000 7fde270e4640 1 -- start start 2024-03-25T19:20:42.786 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.784+0000 7fde270e4640 10 monclient: init 2024-03-25T19:20:42.786 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.785+0000 7fde270e4640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:42.786 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.785+0000 7fde270e4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:20:42.786 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.785+0000 7fde270e4640 10 monclient(hunting): picked mon.noname-a con 0x7fde200b4ff0 addr v1:172.21.15.6:6789/0 2024-03-25T19:20:42.786 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.785+0000 7fde270e4640 10 monclient(hunting): picked mon.noname-b con 0x7fde200b4670 addr v1:172.21.15.6:6790/0 2024-03-25T19:20:42.786 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.785+0000 7fde270e4640 10 monclient(hunting): picked mon.noname-c con 0x7fde2013c480 addr v1:172.21.15.192:6789/0 2024-03-25T19:20:42.786 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.785+0000 7fde270e4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde200ab0e0 con 0x7fde200b4ff0 2024-03-25T19:20:42.786 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.785+0000 7fde270e4640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde200ab3c0 con 0x7fde2013c480 2024-03-25T19:20:42.786 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.785+0000 7fde270e4640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde200a9f30 con 0x7fde200b4670 2024-03-25T19:20:42.786 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.785+0000 7fde270e4640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:42.786 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.785+0000 7fde270e4640 10 monclient(hunting): authenticate will time out at 6347.025373s 2024-03-25T19:20:42.793 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.793+0000 7fde25681640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fde2013c480 0x7fde2014dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:56956/0 (socket says 172.21.15.6:56956) 2024-03-25T19:20:42.793 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.794+0000 7fde25681640 1 -- 172.21.15.6:0/350808470 learned_addr learned my addr 172.21.15.6:0/350808470 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:20:42.806 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.805+0000 7fde1f7fe640 1 -- 172.21.15.6:0/350808470 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3235909999 0 0) 0x7fde200ab3c0 con 0x7fde2013c480 2024-03-25T19:20:42.806 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.805+0000 7fde1f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:20:42.806 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.805+0000 7fde1f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:42.806 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.805+0000 7fde1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:42.806 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.805+0000 7fde1f7fe640 10 monclient(hunting): my global_id is 4525 2024-03-25T19:20:42.806 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.805+0000 7fde1f7fe640 1 -- 172.21.15.6:0/350808470 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde10003520 con 0x7fde2013c480 2024-03-25T19:20:42.810 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.810+0000 7fde1f7fe640 1 -- 172.21.15.6:0/350808470 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 174410690 0 0) 0x7fde200ab0e0 con 0x7fde200b4ff0 2024-03-25T19:20:42.810 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.810+0000 7fde1f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:20:42.810 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.810+0000 7fde1f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:42.810 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.810+0000 7fde1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:42.810 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.810+0000 7fde1f7fe640 10 monclient(hunting): my global_id is 4521 2024-03-25T19:20:42.810 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.810+0000 7fde1f7fe640 1 -- 172.21.15.6:0/350808470 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde200ab3c0 con 0x7fde200b4ff0 2024-03-25T19:20:42.817 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.817+0000 7fde1f7fe640 1 -- 172.21.15.6:0/350808470 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2754200031 0 0) 0x7fde200a9f30 con 0x7fde200b4670 2024-03-25T19:20:42.817 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.817+0000 7fde1f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:20:42.817 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.817+0000 7fde1f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:42.817 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.817+0000 7fde1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:42.817 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.817+0000 7fde1f7fe640 10 monclient(hunting): my global_id is 4520 2024-03-25T19:20:42.817 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.817+0000 7fde1f7fe640 1 -- 172.21.15.6:0/350808470 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde200ab0e0 con 0x7fde200b4670 2024-03-25T19:20:42.818 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.818+0000 7fde1f7fe640 1 -- 172.21.15.6:0/350808470 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1600853292 0 0) 0x7fde200ab3c0 con 0x7fde200b4ff0 2024-03-25T19:20:42.818 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.818+0000 7fde1f7fe640 1 -- 172.21.15.6:0/350808470 >> v1:172.21.15.6:6790/0 conn(0x7fde200b4670 legacy=0x7fde201425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:42.818 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.818+0000 7fde1f7fe640 1 -- 172.21.15.6:0/350808470 >> v1:172.21.15.192:6789/0 conn(0x7fde2013c480 legacy=0x7fde2014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:42.818 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.818+0000 7fde1f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:20:42.819 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.818+0000 7fde1f7fe640 1 monclient: found mon.noname-a 2024-03-25T19:20:42.819 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.818+0000 7fde1f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:42.819 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.818+0000 7fde1f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-03-25T19:20:42.819 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.818+0000 7fde1f7fe640 1 -- 172.21.15.6:0/350808470 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde200a9b70 con 0x7fde200b4ff0 2024-03-25T19:20:42.819 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.818+0000 7fde1f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:20:42.819 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.818+0000 7fde1f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:20:42.819 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.818+0000 7fde1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:42.819 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.818+0000 7fde1f7fe640 1 -- 172.21.15.6:0/350808470 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fde14001670 con 0x7fde200b4ff0 2024-03-25T19:20:42.819 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.818+0000 7fde1f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:42.819 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.818+0000 7fde1f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:20:42.819 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.818+0000 7fde1f7fe640 10 monclient: dump: 2024-03-25T19:20:42.819 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:20:42.819 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:42.819 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:42.819 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:42.820 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:42.820 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:42.820 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:20:42.820 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:20:42.820 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:20:42.820 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:42.820 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.818+0000 7fde270e4640 5 monclient: authenticate success, global_id 4521 2024-03-25T19:20:42.820 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.818+0000 7fde270e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:42.836 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.835+0000 7fde1f7fe640 1 -- 172.21.15.6:0/350808470 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde140017d0 con 0x7fde200b4ff0 2024-03-25T19:20:42.836 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.835+0000 7fde1f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:42.836 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.835+0000 7fde1f7fe640 1 -- 172.21.15.6:0/350808470 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fde14001be0 con 0x7fde200b4ff0 2024-03-25T19:20:42.836 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.835+0000 7fde270e4640 10 monclient: get_monmap_and_config success 2024-03-25T19:20:42.836 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.835+0000 7fde270e4640 4 set_mon_vals no callback set 2024-03-25T19:20:42.836 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.835+0000 7fde1f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:42.836 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.835+0000 7fde1f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:20:42.836 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.835+0000 7fde1f7fe640 10 monclient: dump: 2024-03-25T19:20:42.836 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:20:42.836 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:42.836 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:42.836 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:42.836 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:42.837 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:42.837 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:20:42.837 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:20:42.837 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:20:42.837 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:42.839 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.836+0000 7fde270e4640 10 monclient: shutdown 2024-03-25T19:20:42.839 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.836+0000 7fde270e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:42.839 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.836+0000 7fde270e4640 1 -- 172.21.15.6:0/350808470 >> v1:172.21.15.6:6789/0 conn(0x7fde200b4ff0 legacy=0x7fde2013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:42.839 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.836+0000 7fde270e4640 1 -- 172.21.15.6:0/350808470 shutdown_connections 2024-03-25T19:20:42.839 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.836+0000 7fde270e4640 1 -- 172.21.15.6:0/350808470 >> 172.21.15.6:0/350808470 conn(0x7fde200617a0 msgr2=0x7fde20061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:42.839 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.836+0000 7fde270e4640 1 -- 172.21.15.6:0/350808470 shutdown_connections 2024-03-25T19:20:42.839 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.836+0000 7fde270e4640 1 -- 172.21.15.6:0/350808470 wait complete. 2024-03-25T19:20:42.839 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.837+0000 7fde270e4640 10 monclient: build_initial_monmap 2024-03-25T19:20:42.839 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.837+0000 7fde270e4640 10 monclient: monmap: 2024-03-25T19:20:42.839 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:20:42.839 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:20:42.840 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:20:42.840 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:20:42.840 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:42.840 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:20:42.840 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:20:42.840 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:20:42.840 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:20:42.840 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:42.840 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.837+0000 7fde270e4640 1 librados: starting msgr at 2024-03-25T19:20:42.840 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.837+0000 7fde270e4640 1 librados: starting objecter 2024-03-25T19:20:42.840 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.838+0000 7fde270e4640 1 Processor -- start 2024-03-25T19:20:42.840 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.838+0000 7fde270e4640 1 -- start start 2024-03-25T19:20:42.840 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.838+0000 7fde270e4640 1 librados: setting wanted keys 2024-03-25T19:20:42.840 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.838+0000 7fde270e4640 1 librados: calling monclient init 2024-03-25T19:20:42.840 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.838+0000 7fde270e4640 10 monclient: init 2024-03-25T19:20:42.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.838+0000 7fde270e4640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:42.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.838+0000 7fde270e4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:20:42.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.838+0000 7fde270e4640 10 monclient(hunting): picked mon.noname-c con 0x7fde200b4670 addr v1:172.21.15.6:6790/0 2024-03-25T19:20:42.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.838+0000 7fde270e4640 10 monclient(hunting): picked mon.noname-b con 0x7fde200b4ff0 addr v1:172.21.15.192:6789/0 2024-03-25T19:20:42.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.838+0000 7fde270e4640 10 monclient(hunting): picked mon.noname-a con 0x7fde2013c480 addr v1:172.21.15.6:6789/0 2024-03-25T19:20:42.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.838+0000 7fde270e4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde20130c60 con 0x7fde2013c480 2024-03-25T19:20:42.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.839+0000 7fde270e4640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde2013e900 con 0x7fde200b4ff0 2024-03-25T19:20:42.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.839+0000 7fde270e4640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde2013e200 con 0x7fde200b4670 2024-03-25T19:20:42.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.839+0000 7fde270e4640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:42.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.839+0000 7fde270e4640 10 monclient(hunting): authenticate will time out at 6347.078805s 2024-03-25T19:20:42.848 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.848+0000 7fde1ffff640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fde200b4ff0 0x7fde2012fe40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:56988/0 (socket says 172.21.15.6:56988) 2024-03-25T19:20:42.848 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.848+0000 7fde1ffff640 1 -- 172.21.15.6:0/483334229 learned_addr learned my addr 172.21.15.6:0/483334229 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:20:42.848 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.848+0000 7fde1d7fa640 10 client.?.objecter ms_handle_connect 0x7fde200b4ff0 2024-03-25T19:20:42.848 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.848+0000 7fde1d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:42.852 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.852+0000 7fde1d7fa640 10 client.?.objecter ms_handle_connect 0x7fde200b4670 2024-03-25T19:20:42.852 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.852+0000 7fde1d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:42.865 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.864+0000 7fde1d7fa640 1 -- 172.21.15.6:0/483334229 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 661322660 0 0) 0x7fde2013e900 con 0x7fde200b4ff0 2024-03-25T19:20:42.865 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.865+0000 7fde1d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:20:42.865 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.865+0000 7fde1d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:42.865 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.865+0000 7fde1d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:42.865 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.865+0000 7fde1d7fa640 10 monclient(hunting): my global_id is 4528 2024-03-25T19:20:42.865 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.865+0000 7fde1d7fa640 1 -- 172.21.15.6:0/483334229 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde04004eb0 con 0x7fde200b4ff0 2024-03-25T19:20:42.867 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.867+0000 7fde1d7fa640 10 client.?.objecter ms_handle_connect 0x7fde2013c480 2024-03-25T19:20:42.868 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.867+0000 7fde1d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:42.876 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.876+0000 7fde1d7fa640 1 -- 172.21.15.6:0/483334229 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1692381321 0 0) 0x7fde2013e200 con 0x7fde200b4670 2024-03-25T19:20:42.876 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.876+0000 7fde1d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:20:42.876 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.876+0000 7fde1d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:42.876 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.876+0000 7fde1d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:42.876 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.876+0000 7fde1d7fa640 10 monclient(hunting): my global_id is 4523 2024-03-25T19:20:42.876 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.876+0000 7fde1d7fa640 1 -- 172.21.15.6:0/483334229 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde2013e900 con 0x7fde200b4670 2024-03-25T19:20:42.884 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.884+0000 7fde1d7fa640 1 -- 172.21.15.6:0/483334229 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 902680582 0 0) 0x7fde20130c60 con 0x7fde2013c480 2024-03-25T19:20:42.884 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.884+0000 7fde1d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:20:42.884 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.884+0000 7fde1d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:42.884 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.884+0000 7fde1d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:42.884 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.884+0000 7fde1d7fa640 10 monclient(hunting): my global_id is 4524 2024-03-25T19:20:42.884 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.884+0000 7fde1d7fa640 1 -- 172.21.15.6:0/483334229 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde2013e200 con 0x7fde2013c480 2024-03-25T19:20:42.893 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.893+0000 7fde1d7fa640 1 -- 172.21.15.6:0/483334229 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2548082747 0 0) 0x7fde04004eb0 con 0x7fde200b4ff0 2024-03-25T19:20:42.893 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.893+0000 7fde1d7fa640 1 -- 172.21.15.6:0/483334229 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde20130c60 con 0x7fde200b4ff0 2024-03-25T19:20:42.893 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.893+0000 7fde1d7fa640 1 -- 172.21.15.6:0/483334229 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fde0c001a10 con 0x7fde200b4ff0 2024-03-25T19:20:42.893 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.893+0000 7fde1d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:42.893 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.893+0000 7fde1d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:20:42.893 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.893+0000 7fde1d7fa640 10 monclient(hunting): dump: 2024-03-25T19:20:42.893 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:20:42.893 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:42.893 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:42.894 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:42.894 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:42.894 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:42.894 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:20:42.894 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:20:42.894 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:20:42.894 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:42.894 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.893+0000 7fde1d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:20:42.896 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.895+0000 7fde1d7fa640 1 -- 172.21.15.6:0/483334229 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3018557415 0 0) 0x7fde2013e900 con 0x7fde200b4670 2024-03-25T19:20:42.896 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.895+0000 7fde1d7fa640 1 -- 172.21.15.6:0/483334229 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde04004eb0 con 0x7fde200b4670 2024-03-25T19:20:42.896 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.895+0000 7fde1d7fa640 1 -- 172.21.15.6:0/483334229 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fde14001ae0 con 0x7fde200b4670 2024-03-25T19:20:42.896 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.895+0000 7fde1d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:42.896 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.895+0000 7fde1d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:20:42.896 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.895+0000 7fde1d7fa640 10 monclient(hunting): dump: 2024-03-25T19:20:42.896 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:20:42.896 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:42.896 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:42.896 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:42.897 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:42.897 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:42.897 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:20:42.897 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:20:42.897 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:20:42.897 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:42.907 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.906+0000 7fde1d7fa640 1 -- 172.21.15.6:0/483334229 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 573436501 0 0) 0x7fde20130c60 con 0x7fde200b4ff0 2024-03-25T19:20:42.907 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.906+0000 7fde1d7fa640 1 -- 172.21.15.6:0/483334229 >> v1:172.21.15.6:6790/0 conn(0x7fde200b4670 legacy=0x7fde2013f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:42.907 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.906+0000 7fde1d7fa640 1 -- 172.21.15.6:0/483334229 >> v1:172.21.15.6:6789/0 conn(0x7fde2013c480 legacy=0x7fde20130550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:42.907 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.906+0000 7fde1d7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:20:42.907 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.906+0000 7fde1d7fa640 1 monclient: found mon.b 2024-03-25T19:20:42.907 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.906+0000 7fde1d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:42.907 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.906+0000 7fde1d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:20:42.907 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.906+0000 7fde1d7fa640 1 -- 172.21.15.6:0/483334229 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde2013e4e0 con 0x7fde200b4ff0 2024-03-25T19:20:42.907 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.906+0000 7fde1d7fa640 10 monclient: _finish_auth 0 2024-03-25T19:20:42.908 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.906+0000 7fde1d7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:20:42.908 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.906+0000 7fde1d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:42.908 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.906+0000 7fde270e4640 5 monclient: authenticate success, global_id 4528 2024-03-25T19:20:42.908 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.906+0000 7fde270e4640 10 monclient: _renew_subs 2024-03-25T19:20:42.908 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.906+0000 7fde270e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:20:42.908 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.906+0000 7fde270e4640 1 -- 172.21.15.6:0/483334229 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde20145e60 con 0x7fde200b4ff0 2024-03-25T19:20:42.908 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.907+0000 7fde270e4640 10 client.4528.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:20:42.908 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.907+0000 7fde270e4640 10 monclient: _renew_subs 2024-03-25T19:20:42.908 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.907+0000 7fde270e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:20:42.908 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.907+0000 7fde270e4640 1 -- 172.21.15.6:0/483334229 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fde20143a80 con 0x7fde200b4ff0 2024-03-25T19:20:42.908 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.907+0000 7fde270e4640 1 librados: init done 2024-03-25T19:20:42.909 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.908+0000 7fde270e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:42.909 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.908+0000 7fde270e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:42.909 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.908+0000 7fde270e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:20:42.909 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:42.908+0000 7fde270e4640 1 -- 172.21.15.6:0/483334229 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde2006dbd0 con 0x7fde200b4ff0 2024-03-25T19:20:43.224 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:43.224+0000 7fde1d7fa640 1 -- 172.21.15.6:0/483334229 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde0c001c70 con 0x7fde200b4ff0 2024-03-25T19:20:43.225 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:43.224+0000 7fde1d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:43.225 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:43.224+0000 7fde1d7fa640 1 -- 172.21.15.6:0/483334229 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fde0c003a60 con 0x7fde200b4ff0 2024-03-25T19:20:43.225 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:43.224+0000 7fde1d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:43.225 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:43.224+0000 7fde1d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:20:43.225 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:43.224+0000 7fde1d7fa640 10 monclient: dump: 2024-03-25T19:20:43.225 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:20:43.225 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:43.225 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:43.225 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:43.225 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:43.225 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:43.225 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:20:43.225 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:20:43.225 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:20:43.226 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:43.226 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:43.224+0000 7fde1effd640 4 set_mon_vals no callback set 2024-03-25T19:20:43.226 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:43.225+0000 7fde1d7fa640 1 -- 172.21.15.6:0/483334229 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7fde0c003c80 con 0x7fde200b4ff0 2024-03-25T19:20:43.226 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:43.226+0000 7fde24e80640 10 monclient: get_auth_request con 0x7fde0406df70 auth_method 0 2024-03-25T19:20:43.226 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:43.226+0000 7fde1d7fa640 10 client.4528.objecter ms_handle_connect 0x7fde0406df70 2024-03-25T19:20:43.229 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:43.229+0000 7fde1d7fa640 1 -- 172.21.15.6:0/483334229 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7fde0c0548b0 con 0x7fde200b4ff0 2024-03-25T19:20:43.229 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:43.229+0000 7fde1d7fa640 10 client.4528.objecter ms_dispatch 0x7fde20000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:20:43.229 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:43.229+0000 7fde1d7fa640 3 client.4528.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:20:43.229 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:43.229+0000 7fde1d7fa640 3 client.4528.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:20:43.229 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:43.229+0000 7fde1d7fa640 20 client.4528.objecter dump_active .. 0 homeless 2024-03-25T19:20:43.230 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:43.230+0000 7fde1d7fa640 1 -- 172.21.15.6:0/483334229 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fde0c0503c0 con 0x7fde200b4ff0 2024-03-25T19:20:43.230 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:43.230+0000 7fde1d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:43.230 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:43.230+0000 7fde1d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:20:43.439 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:43.439+0000 7fde270e4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:43.439 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:43.439+0000 7fde270e4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:43.439 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:43.439+0000 7fde270e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:20:43.439 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:43.439+0000 7fde270e4640 1 -- 172.21.15.6:0/483334229 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fde2014f100 con 0x7fde200b4ff0 2024-03-25T19:20:43.514 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:43.514+0000 7fde1d7fa640 1 -- 172.21.15.6:0/483334229 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7fde0c0522f0 con 0x7fde200b4ff0 2024-03-25T19:20:43.515 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:43.514+0000 7fde1d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:43.515 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:43.514+0000 7fde1d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:20:43.515 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:20:43.515 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:20:43.517 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:43.517+0000 7fde270e4640 10 librados: watch_flush enter 2024-03-25T19:20:43.517 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:43.517+0000 7fde270e4640 10 librados: watch_flush exit 2024-03-25T19:20:43.517 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:43.517+0000 7fde270e4640 20 client.4528.objecter shutdown clearing up homeless session... 2024-03-25T19:20:43.517 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:43.517+0000 7fde270e4640 10 client.4528.objecter successfully canceled tick 2024-03-25T19:20:43.518 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:43.517+0000 7fde270e4640 1 -- 172.21.15.6:0/483334229 >> v1:172.21.15.6:6800/2936652650 conn(0x7fde0406df70 legacy=0x7fde04070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:43.518 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:43.517+0000 7fde270e4640 10 monclient: shutdown 2024-03-25T19:20:43.518 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:43.517+0000 7fde270e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:43.518 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:43.517+0000 7fde270e4640 1 -- 172.21.15.6:0/483334229 >> v1:172.21.15.192:6789/0 conn(0x7fde200b4ff0 legacy=0x7fde2012fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:43.518 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:43.517+0000 7fde270e4640 1 -- 172.21.15.6:0/483334229 shutdown_connections 2024-03-25T19:20:43.518 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:43.517+0000 7fde270e4640 1 -- 172.21.15.6:0/483334229 >> 172.21.15.6:0/483334229 conn(0x7fde200617a0 msgr2=0x7fde20140670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:43.518 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:43.517+0000 7fde270e4640 1 -- 172.21.15.6:0/483334229 shutdown_connections 2024-03-25T19:20:43.518 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:43.517+0000 7fde270e4640 1 -- 172.21.15.6:0/483334229 wait complete. 2024-03-25T19:20:43.518 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:43.517+0000 7fde270e4640 1 librados: shutdown 2024-03-25T19:20:43.529 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:20:48.800 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:07:06.147 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:48.804 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:07:06.151 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:48.836 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:07:06.183 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:48.857 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:07:06.174 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:48.862 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:07:06.180 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:48.871 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:07:06.218 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:48.875 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:07:06.222 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:48.892 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:07:06.210 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:48.914 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:07:06.261 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:50.336 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:20:50.450 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.450+0000 7f4f1ad64640 10 monclient: get_monmap_and_config 2024-03-25T19:20:50.450 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.450+0000 7f4f1ad64640 10 monclient: build_initial_monmap 2024-03-25T19:20:50.450 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.450+0000 7f4f1ad64640 10 monclient: monmap: 2024-03-25T19:20:50.450 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:20:50.450 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:50.450 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:20:50.451102+0000 2024-03-25T19:20:50.451 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:20:50.451102+0000 2024-03-25T19:20:50.451 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:50.451 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:50.451 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:20:50.451 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:20:50.451 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:20:50.451 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:50.452 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.451+0000 7f4f1ad64640 1 Processor -- start 2024-03-25T19:20:50.452 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.451+0000 7f4f1ad64640 1 -- start start 2024-03-25T19:20:50.452 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.451+0000 7f4f1ad64640 10 monclient: init 2024-03-25T19:20:50.452 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.451+0000 7f4f1ad64640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:50.452 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.451+0000 7f4f1ad64640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:20:50.452 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.451+0000 7f4f1ad64640 10 monclient(hunting): picked mon.noname-b con 0x7f4f140ae940 addr v1:172.21.15.6:6790/0 2024-03-25T19:20:50.453 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.451+0000 7f4f1ad64640 10 monclient(hunting): picked mon.noname-a con 0x7f4f14139070 addr v1:172.21.15.6:6789/0 2024-03-25T19:20:50.453 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.451+0000 7f4f1ad64640 10 monclient(hunting): picked mon.noname-c con 0x7f4f140706e0 addr v1:172.21.15.192:6789/0 2024-03-25T19:20:50.453 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.451+0000 7f4f1ad64640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f14071880 con 0x7f4f14139070 2024-03-25T19:20:50.453 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.451+0000 7f4f1ad64640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f14071ab0 con 0x7f4f140706e0 2024-03-25T19:20:50.453 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.451+0000 7f4f1ad64640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f1413cb10 con 0x7f4f140ae940 2024-03-25T19:20:50.453 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.451+0000 7f4f1ad64640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:50.453 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.451+0000 7f4f1ad64640 10 monclient(hunting): authenticate will time out at 6354.691735s 2024-03-25T19:20:50.462 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.462+0000 7f4f19301640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f4f140706e0 0x7f4f14072730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:59586/0 (socket says 172.21.15.6:59586) 2024-03-25T19:20:50.462 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.462+0000 7f4f19301640 1 -- 172.21.15.6:0/2928467640 learned_addr learned my addr 172.21.15.6:0/2928467640 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:20:50.478 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.478+0000 7f4f137fe640 1 -- 172.21.15.6:0/2928467640 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3135879878 0 0) 0x7f4f14071ab0 con 0x7f4f140706e0 2024-03-25T19:20:50.478 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.478+0000 7f4f137fe640 10 monclient(hunting): _negotiate 2024-03-25T19:20:50.478 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.478+0000 7f4f137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:50.478 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.478+0000 7f4f137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:50.478 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.478+0000 7f4f137fe640 10 monclient(hunting): my global_id is 4531 2024-03-25T19:20:50.478 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.478+0000 7f4f137fe640 1 -- 172.21.15.6:0/2928467640 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f04003520 con 0x7f4f140706e0 2024-03-25T19:20:50.487 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.487+0000 7f4f137fe640 1 -- 172.21.15.6:0/2928467640 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3367132479 0 0) 0x7f4f14071880 con 0x7f4f14139070 2024-03-25T19:20:50.487 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.487+0000 7f4f137fe640 10 monclient(hunting): _negotiate 2024-03-25T19:20:50.487 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.487+0000 7f4f137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:50.487 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.487+0000 7f4f137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:50.487 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.487+0000 7f4f137fe640 10 monclient(hunting): my global_id is 4527 2024-03-25T19:20:50.487 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.487+0000 7f4f137fe640 1 -- 172.21.15.6:0/2928467640 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f14071ab0 con 0x7f4f14139070 2024-03-25T19:20:50.502 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.501+0000 7f4f137fe640 1 -- 172.21.15.6:0/2928467640 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3014507570 0 0) 0x7f4f1413cb10 con 0x7f4f140ae940 2024-03-25T19:20:50.502 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.501+0000 7f4f137fe640 10 monclient(hunting): _negotiate 2024-03-25T19:20:50.502 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.501+0000 7f4f137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:50.502 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.501+0000 7f4f137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:50.502 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.501+0000 7f4f137fe640 10 monclient(hunting): my global_id is 4526 2024-03-25T19:20:50.503 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.501+0000 7f4f137fe640 1 -- 172.21.15.6:0/2928467640 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f14071880 con 0x7f4f140ae940 2024-03-25T19:20:50.503 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.502+0000 7f4f137fe640 1 -- 172.21.15.6:0/2928467640 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2277975745 0 0) 0x7f4f14071ab0 con 0x7f4f14139070 2024-03-25T19:20:50.503 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.502+0000 7f4f137fe640 1 -- 172.21.15.6:0/2928467640 >> v1:172.21.15.6:6790/0 conn(0x7f4f140ae940 legacy=0x7f4f14138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:50.503 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.502+0000 7f4f137fe640 1 -- 172.21.15.6:0/2928467640 >> v1:172.21.15.192:6789/0 conn(0x7f4f140706e0 legacy=0x7f4f14072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:50.503 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.502+0000 7f4f137fe640 10 monclient: _finish_hunting 0 2024-03-25T19:20:50.503 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.502+0000 7f4f137fe640 1 monclient: found mon.noname-a 2024-03-25T19:20:50.503 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.502+0000 7f4f137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:50.503 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.502+0000 7f4f137fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-03-25T19:20:50.503 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.502+0000 7f4f137fe640 1 -- 172.21.15.6:0/2928467640 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f1406fa30 con 0x7f4f14139070 2024-03-25T19:20:50.503 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.502+0000 7f4f137fe640 10 monclient: _finish_auth 0 2024-03-25T19:20:50.503 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.502+0000 7f4f137fe640 10 monclient: _check_auth_tickets 2024-03-25T19:20:50.503 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.502+0000 7f4f137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:50.503 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.502+0000 7f4f137fe640 1 -- 172.21.15.6:0/2928467640 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f4f00001670 con 0x7f4f14139070 2024-03-25T19:20:50.504 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.502+0000 7f4f1ad64640 5 monclient: authenticate success, global_id 4527 2024-03-25T19:20:50.504 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.502+0000 7f4f1ad64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:50.504 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.502+0000 7f4f137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:50.504 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.502+0000 7f4f137fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:20:50.504 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.502+0000 7f4f137fe640 10 monclient: dump: 2024-03-25T19:20:50.504 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:20:50.504 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:50.504 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:50.504 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:50.504 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:50.504 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:50.504 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:20:50.504 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:20:50.504 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:20:50.505 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:50.505 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.502+0000 7f4f1ad64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:50.524 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.524+0000 7f4f137fe640 1 -- 172.21.15.6:0/2928467640 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f000017d0 con 0x7f4f14139070 2024-03-25T19:20:50.524 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.524+0000 7f4f137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:50.524 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.524+0000 7f4f137fe640 1 -- 172.21.15.6:0/2928467640 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f4f00001b60 con 0x7f4f14139070 2024-03-25T19:20:50.524 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.524+0000 7f4f1ad64640 10 monclient: get_monmap_and_config success 2024-03-25T19:20:50.524 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.524+0000 7f4f1ad64640 4 set_mon_vals no callback set 2024-03-25T19:20:50.524 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.524+0000 7f4f137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:50.525 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.524+0000 7f4f137fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:20:50.525 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.524+0000 7f4f137fe640 10 monclient: dump: 2024-03-25T19:20:50.525 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:20:50.525 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:50.525 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:50.525 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:50.525 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:50.525 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:50.525 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:20:50.525 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:20:50.525 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:20:50.525 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:50.527 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.525+0000 7f4f1ad64640 10 monclient: shutdown 2024-03-25T19:20:50.527 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.525+0000 7f4f1ad64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:50.527 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.525+0000 7f4f1ad64640 1 -- 172.21.15.6:0/2928467640 >> v1:172.21.15.6:6789/0 conn(0x7f4f14139070 legacy=0x7f4f14072020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:50.528 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.525+0000 7f4f1ad64640 1 -- 172.21.15.6:0/2928467640 shutdown_connections 2024-03-25T19:20:50.528 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.525+0000 7f4f1ad64640 1 -- 172.21.15.6:0/2928467640 >> 172.21.15.6:0/2928467640 conn(0x7f4f140b3cd0 msgr2=0x7f4f140b40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:50.528 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.525+0000 7f4f1ad64640 1 -- 172.21.15.6:0/2928467640 shutdown_connections 2024-03-25T19:20:50.528 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.525+0000 7f4f1ad64640 1 -- 172.21.15.6:0/2928467640 wait complete. 2024-03-25T19:20:50.528 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.526+0000 7f4f1ad64640 10 monclient: build_initial_monmap 2024-03-25T19:20:50.528 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.526+0000 7f4f1ad64640 10 monclient: monmap: 2024-03-25T19:20:50.528 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:20:50.528 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:20:50.528 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:20:50.528 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:20:50.528 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:50.528 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:20:50.528 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:20:50.528 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:20:50.528 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:20:50.529 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:50.529 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.526+0000 7f4f1ad64640 1 librados: starting msgr at 2024-03-25T19:20:50.529 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.526+0000 7f4f1ad64640 1 librados: starting objecter 2024-03-25T19:20:50.529 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.526+0000 7f4f1ad64640 1 Processor -- start 2024-03-25T19:20:50.529 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.526+0000 7f4f1ad64640 1 -- start start 2024-03-25T19:20:50.529 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.526+0000 7f4f1ad64640 1 librados: setting wanted keys 2024-03-25T19:20:50.529 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.526+0000 7f4f1ad64640 1 librados: calling monclient init 2024-03-25T19:20:50.529 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.526+0000 7f4f1ad64640 10 monclient: init 2024-03-25T19:20:50.529 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.527+0000 7f4f1ad64640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:50.529 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.527+0000 7f4f1ad64640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:20:50.529 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.527+0000 7f4f1ad64640 10 monclient(hunting): picked mon.noname-c con 0x7f4f140706e0 addr v1:172.21.15.6:6790/0 2024-03-25T19:20:50.529 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.527+0000 7f4f1ad64640 10 monclient(hunting): picked mon.noname-a con 0x7f4f140ae940 addr v1:172.21.15.6:6789/0 2024-03-25T19:20:50.529 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.527+0000 7f4f1ad64640 10 monclient(hunting): picked mon.noname-b con 0x7f4f14139070 addr v1:172.21.15.192:6789/0 2024-03-25T19:20:50.529 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.527+0000 7f4f1ad64640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f141395a0 con 0x7f4f140ae940 2024-03-25T19:20:50.530 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.527+0000 7f4f1ad64640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f1413cdd0 con 0x7f4f14139070 2024-03-25T19:20:50.530 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.527+0000 7f4f1ad64640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f1413aa80 con 0x7f4f140706e0 2024-03-25T19:20:50.530 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.527+0000 7f4f1ad64640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:50.530 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.527+0000 7f4f1ad64640 10 monclient(hunting): authenticate will time out at 6354.767269s 2024-03-25T19:20:50.532 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.533+0000 7f4f18b00640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f4f140706e0 0x7f4f14139ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:41136/0 (socket says 172.21.15.6:41136) 2024-03-25T19:20:50.533 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.533+0000 7f4f18b00640 1 -- 172.21.15.6:0/747904198 learned_addr learned my addr 172.21.15.6:0/747904198 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:20:50.533 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.533+0000 7f4f117fa640 10 client.?.objecter ms_handle_connect 0x7f4f140706e0 2024-03-25T19:20:50.533 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.534+0000 7f4f117fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:50.541 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.541+0000 7f4f117fa640 10 client.?.objecter ms_handle_connect 0x7f4f14139070 2024-03-25T19:20:50.541 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.541+0000 7f4f117fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:50.549 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.549+0000 7f4f117fa640 10 client.?.objecter ms_handle_connect 0x7f4f140ae940 2024-03-25T19:20:50.549 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.549+0000 7f4f117fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:50.550 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.550+0000 7f4f117fa640 1 -- 172.21.15.6:0/747904198 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1650416016 0 0) 0x7f4f1413cdd0 con 0x7f4f14139070 2024-03-25T19:20:50.550 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.550+0000 7f4f117fa640 10 monclient(hunting): _negotiate 2024-03-25T19:20:50.550 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.550+0000 7f4f117fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:50.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.550+0000 7f4f117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:50.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.550+0000 7f4f117fa640 10 monclient(hunting): my global_id is 4534 2024-03-25T19:20:50.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.550+0000 7f4f117fa640 1 -- 172.21.15.6:0/747904198 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ef8004eb0 con 0x7f4f14139070 2024-03-25T19:20:50.553 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.553+0000 7f4f117fa640 1 -- 172.21.15.6:0/747904198 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 909764531 0 0) 0x7f4f1413aa80 con 0x7f4f140706e0 2024-03-25T19:20:50.553 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.553+0000 7f4f117fa640 10 monclient(hunting): _negotiate 2024-03-25T19:20:50.553 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.553+0000 7f4f117fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:50.553 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.553+0000 7f4f117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:50.553 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.553+0000 7f4f117fa640 10 monclient(hunting): my global_id is 4529 2024-03-25T19:20:50.553 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.553+0000 7f4f117fa640 1 -- 172.21.15.6:0/747904198 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f1413cdd0 con 0x7f4f140706e0 2024-03-25T19:20:50.566 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.566+0000 7f4f117fa640 1 -- 172.21.15.6:0/747904198 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3739950065 0 0) 0x7f4f141395a0 con 0x7f4f140ae940 2024-03-25T19:20:50.566 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.566+0000 7f4f117fa640 10 monclient(hunting): _negotiate 2024-03-25T19:20:50.566 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.566+0000 7f4f117fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:50.566 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.566+0000 7f4f117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:50.566 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.566+0000 7f4f117fa640 10 monclient(hunting): my global_id is 4530 2024-03-25T19:20:50.566 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.566+0000 7f4f117fa640 1 -- 172.21.15.6:0/747904198 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f1413aa80 con 0x7f4f140ae940 2024-03-25T19:20:50.573 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.573+0000 7f4f117fa640 1 -- 172.21.15.6:0/747904198 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4060401753 0 0) 0x7f4ef8004eb0 con 0x7f4f14139070 2024-03-25T19:20:50.573 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.573+0000 7f4f117fa640 1 -- 172.21.15.6:0/747904198 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f141395a0 con 0x7f4f14139070 2024-03-25T19:20:50.573 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.573+0000 7f4f117fa640 1 -- 172.21.15.6:0/747904198 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f4f0c001cd0 con 0x7f4f14139070 2024-03-25T19:20:50.573 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.573+0000 7f4f117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:50.573 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.573+0000 7f4f117fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:20:50.574 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.573+0000 7f4f117fa640 10 monclient(hunting): dump: 2024-03-25T19:20:50.574 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:20:50.574 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:50.574 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:50.574 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:50.574 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:50.574 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:50.574 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:20:50.574 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:20:50.574 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:20:50.574 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:50.574 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.573+0000 7f4f117fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:20:50.582 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.581+0000 7f4f117fa640 1 -- 172.21.15.6:0/747904198 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4206306628 0 0) 0x7f4f141395a0 con 0x7f4f14139070 2024-03-25T19:20:50.582 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.581+0000 7f4f117fa640 1 -- 172.21.15.6:0/747904198 >> v1:172.21.15.6:6790/0 conn(0x7f4f140706e0 legacy=0x7f4f14139ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:50.582 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.581+0000 7f4f117fa640 1 -- 172.21.15.6:0/747904198 >> v1:172.21.15.6:6789/0 conn(0x7f4f140ae940 legacy=0x7f4f1413a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:50.582 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.581+0000 7f4f117fa640 10 monclient: _finish_hunting 0 2024-03-25T19:20:50.582 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.581+0000 7f4f117fa640 1 monclient: found mon.b 2024-03-25T19:20:50.582 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.582+0000 7f4f117fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:50.582 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.582+0000 7f4f117fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:20:50.583 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.582+0000 7f4f117fa640 1 -- 172.21.15.6:0/747904198 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f1413ac10 con 0x7f4f14139070 2024-03-25T19:20:50.583 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.582+0000 7f4f117fa640 10 monclient: _finish_auth 0 2024-03-25T19:20:50.583 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.582+0000 7f4f117fa640 10 monclient: _check_auth_tickets 2024-03-25T19:20:50.583 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.582+0000 7f4f117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:50.583 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.582+0000 7f4f1ad64640 5 monclient: authenticate success, global_id 4534 2024-03-25T19:20:50.583 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.582+0000 7f4f1ad64640 10 monclient: _renew_subs 2024-03-25T19:20:50.583 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.582+0000 7f4f1ad64640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:20:50.583 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.582+0000 7f4f1ad64640 1 -- 172.21.15.6:0/747904198 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f1413b3b0 con 0x7f4f14139070 2024-03-25T19:20:50.583 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.582+0000 7f4f1ad64640 10 client.4534.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:20:50.583 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.582+0000 7f4f1ad64640 10 monclient: _renew_subs 2024-03-25T19:20:50.583 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.582+0000 7f4f1ad64640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:20:50.583 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.582+0000 7f4f1ad64640 1 -- 172.21.15.6:0/747904198 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f140adcb0 con 0x7f4f14139070 2024-03-25T19:20:50.583 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.582+0000 7f4f1ad64640 1 librados: init done 2024-03-25T19:20:50.584 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.583+0000 7f4f1ad64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:50.584 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.583+0000 7f4f1ad64640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:50.584 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.583+0000 7f4f1ad64640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:20:50.584 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.583+0000 7f4f1ad64640 1 -- 172.21.15.6:0/747904198 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f1413b6b0 con 0x7f4f14139070 2024-03-25T19:20:50.862 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.861+0000 7f4f117fa640 1 -- 172.21.15.6:0/747904198 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f0c0020e0 con 0x7f4f14139070 2024-03-25T19:20:50.862 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.861+0000 7f4f117fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:50.862 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.861+0000 7f4f117fa640 1 -- 172.21.15.6:0/747904198 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f4f0c004b60 con 0x7f4f14139070 2024-03-25T19:20:50.862 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.861+0000 7f4f117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:50.862 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.861+0000 7f4f117fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:20:50.862 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.861+0000 7f4f117fa640 10 monclient: dump: 2024-03-25T19:20:50.863 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:20:50.863 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:50.863 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:50.863 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:50.863 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:50.863 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:50.863 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:20:50.863 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:20:50.863 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:20:50.863 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:50.863 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.861+0000 7f4f12ffd640 4 set_mon_vals no callback set 2024-03-25T19:20:50.863 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.862+0000 7f4f117fa640 1 -- 172.21.15.6:0/747904198 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f4f0c004d80 con 0x7f4f14139070 2024-03-25T19:20:50.864 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.863+0000 7f4f18b00640 10 monclient: get_auth_request con 0x7f4ef806ddf0 auth_method 0 2024-03-25T19:20:50.864 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.863+0000 7f4f117fa640 10 client.4534.objecter ms_handle_connect 0x7f4ef806ddf0 2024-03-25T19:20:50.867 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.867+0000 7f4f117fa640 1 -- 172.21.15.6:0/747904198 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f4f0c054930 con 0x7f4f14139070 2024-03-25T19:20:50.868 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.867+0000 7f4f117fa640 10 client.4534.objecter ms_dispatch 0x7f4f14000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:20:50.868 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.867+0000 7f4f117fa640 3 client.4534.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:20:50.868 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.867+0000 7f4f117fa640 3 client.4534.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:20:50.868 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.868+0000 7f4f117fa640 20 client.4534.objecter dump_active .. 0 homeless 2024-03-25T19:20:50.868 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.868+0000 7f4f117fa640 1 -- 172.21.15.6:0/747904198 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4f0c050270 con 0x7f4f14139070 2024-03-25T19:20:50.868 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.868+0000 7f4f117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:50.868 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:50.868+0000 7f4f117fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:20:51.085 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:51.085+0000 7f4f1ad64640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:51.085 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:51.085+0000 7f4f1ad64640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:51.085 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:51.085+0000 7f4f1ad64640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:20:51.085 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:51.085+0000 7f4f1ad64640 1 -- 172.21.15.6:0/747904198 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f1413d200 con 0x7f4f14139070 2024-03-25T19:20:51.123 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:51.123+0000 7f4f117fa640 1 -- 172.21.15.6:0/747904198 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f4f0c052370 con 0x7f4f14139070 2024-03-25T19:20:51.123 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:51.123+0000 7f4f117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:51.124 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:51.123+0000 7f4f117fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:20:51.124 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:20:51.124 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:20:51.127 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:51.125+0000 7f4f1ad64640 10 librados: watch_flush enter 2024-03-25T19:20:51.127 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:51.125+0000 7f4f1ad64640 10 librados: watch_flush exit 2024-03-25T19:20:51.127 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:51.125+0000 7f4f1ad64640 20 client.4534.objecter shutdown clearing up homeless session... 2024-03-25T19:20:51.127 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:51.125+0000 7f4f1ad64640 10 client.4534.objecter successfully canceled tick 2024-03-25T19:20:51.127 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:51.125+0000 7f4f1ad64640 1 -- 172.21.15.6:0/747904198 >> v1:172.21.15.6:6800/2936652650 conn(0x7f4ef806ddf0 legacy=0x7f4ef80702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:51.127 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:51.125+0000 7f4f1ad64640 10 monclient: shutdown 2024-03-25T19:20:51.128 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:51.125+0000 7f4f1ad64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:51.128 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:51.125+0000 7f4f1ad64640 1 -- 172.21.15.6:0/747904198 >> v1:172.21.15.192:6789/0 conn(0x7f4f14139070 legacy=0x7f4f141e1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:51.128 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:51.125+0000 7f4f1ad64640 1 -- 172.21.15.6:0/747904198 shutdown_connections 2024-03-25T19:20:51.128 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:51.125+0000 7f4f1ad64640 1 -- 172.21.15.6:0/747904198 >> 172.21.15.6:0/747904198 conn(0x7f4f140b3cd0 msgr2=0x7f4f1406dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:51.128 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:51.126+0000 7f4f1ad64640 1 -- 172.21.15.6:0/747904198 shutdown_connections 2024-03-25T19:20:51.128 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:51.126+0000 7f4f1ad64640 1 -- 172.21.15.6:0/747904198 wait complete. 2024-03-25T19:20:51.128 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:51.126+0000 7f4f1ad64640 1 librados: shutdown 2024-03-25T19:20:51.139 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:20:57.945 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:20:58.062 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.062+0000 7fe9d0c4a640 10 monclient: get_monmap_and_config 2024-03-25T19:20:58.062 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.062+0000 7fe9d0c4a640 10 monclient: build_initial_monmap 2024-03-25T19:20:58.063 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.062+0000 7fe9d0c4a640 10 monclient: monmap: 2024-03-25T19:20:58.063 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:20:58.063 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:58.063 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:20:58.063810+0000 2024-03-25T19:20:58.063 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:20:58.063810+0000 2024-03-25T19:20:58.063 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:58.063 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:58.063 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:20:58.063 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:20:58.063 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:20:58.063 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:58.064 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.063+0000 7fe9d0c4a640 1 Processor -- start 2024-03-25T19:20:58.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.063+0000 7fe9d0c4a640 1 -- start start 2024-03-25T19:20:58.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.063+0000 7fe9d0c4a640 10 monclient: init 2024-03-25T19:20:58.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.064+0000 7fe9d0c4a640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:58.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.064+0000 7fe9d0c4a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:20:58.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.064+0000 7fe9d0c4a640 10 monclient(hunting): picked mon.noname-a con 0x7fe9cc0b4ff0 addr v1:172.21.15.6:6789/0 2024-03-25T19:20:58.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.064+0000 7fe9d0c4a640 10 monclient(hunting): picked mon.noname-c con 0x7fe9cc0b4670 addr v1:172.21.15.192:6789/0 2024-03-25T19:20:58.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.064+0000 7fe9d0c4a640 10 monclient(hunting): picked mon.noname-b con 0x7fe9cc13c480 addr v1:172.21.15.6:6790/0 2024-03-25T19:20:58.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.064+0000 7fe9d0c4a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9cc0ab0e0 con 0x7fe9cc0b4ff0 2024-03-25T19:20:58.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.064+0000 7fe9d0c4a640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9cc0ab3c0 con 0x7fe9cc0b4670 2024-03-25T19:20:58.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.064+0000 7fe9d0c4a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9cc0a9f30 con 0x7fe9cc13c480 2024-03-25T19:20:58.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.064+0000 7fe9d0c4a640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:58.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.064+0000 7fe9d0c4a640 10 monclient(hunting): authenticate will time out at 6362.304327s 2024-03-25T19:20:58.066 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.066+0000 7fe9cad9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fe9cc13c480 0x7fe9cc14dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45702/0 (socket says 172.21.15.6:45702) 2024-03-25T19:20:58.066 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.066+0000 7fe9cad9d640 1 -- 172.21.15.6:0/2496754049 learned_addr learned my addr 172.21.15.6:0/2496754049 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:20:58.078 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.078+0000 7fe9c9d9b640 1 -- 172.21.15.6:0/2496754049 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1312533558 0 0) 0x7fe9cc0a9f30 con 0x7fe9cc13c480 2024-03-25T19:20:58.078 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.078+0000 7fe9c9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:20:58.078 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.078+0000 7fe9c9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:58.078 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.078+0000 7fe9c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:58.078 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.078+0000 7fe9c9d9b640 10 monclient(hunting): my global_id is 4532 2024-03-25T19:20:58.078 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.078+0000 7fe9c9d9b640 1 -- 172.21.15.6:0/2496754049 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9b8003520 con 0x7fe9cc13c480 2024-03-25T19:20:58.093 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.093+0000 7fe9c9d9b640 1 -- 172.21.15.6:0/2496754049 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 619473873 0 0) 0x7fe9cc0ab3c0 con 0x7fe9cc0b4670 2024-03-25T19:20:58.093 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.093+0000 7fe9c9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:20:58.094 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.093+0000 7fe9c9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:58.094 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.093+0000 7fe9c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:58.094 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.093+0000 7fe9c9d9b640 10 monclient(hunting): my global_id is 4537 2024-03-25T19:20:58.094 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.094+0000 7fe9c9d9b640 1 -- 172.21.15.6:0/2496754049 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9cc0a9f30 con 0x7fe9cc0b4670 2024-03-25T19:20:58.095 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.094+0000 7fe9c9d9b640 1 -- 172.21.15.6:0/2496754049 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 87748084 0 0) 0x7fe9b8003520 con 0x7fe9cc13c480 2024-03-25T19:20:58.095 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.094+0000 7fe9c9d9b640 1 -- 172.21.15.6:0/2496754049 >> v1:172.21.15.192:6789/0 conn(0x7fe9cc0b4670 legacy=0x7fe9cc1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:58.095 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.094+0000 7fe9c9d9b640 1 -- 172.21.15.6:0/2496754049 >> v1:172.21.15.6:6789/0 conn(0x7fe9cc0b4ff0 legacy=0x7fe9cc13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:58.095 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.094+0000 7fe9c9d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:20:58.095 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.094+0000 7fe9c9d9b640 1 monclient: found mon.noname-b 2024-03-25T19:20:58.095 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.094+0000 7fe9c9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:58.095 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.094+0000 7fe9c9d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-03-25T19:20:58.095 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.094+0000 7fe9c9d9b640 1 -- 172.21.15.6:0/2496754049 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9cc0a9b70 con 0x7fe9cc13c480 2024-03-25T19:20:58.095 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.095+0000 7fe9c9d9b640 10 monclient: _finish_auth 0 2024-03-25T19:20:58.095 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.095+0000 7fe9c9d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:20:58.095 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.095+0000 7fe9c9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:58.096 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.095+0000 7fe9c9d9b640 1 -- 172.21.15.6:0/2496754049 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fe9c4001a10 con 0x7fe9cc13c480 2024-03-25T19:20:58.096 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.095+0000 7fe9c9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:58.096 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.095+0000 7fe9c9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:20:58.096 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.095+0000 7fe9c9d9b640 10 monclient: dump: 2024-03-25T19:20:58.096 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:20:58.096 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:58.096 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:58.096 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:58.096 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:58.096 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:58.096 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:20:58.096 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:20:58.096 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:20:58.096 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:58.097 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.095+0000 7fe9d0c4a640 5 monclient: authenticate success, global_id 4532 2024-03-25T19:20:58.097 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.095+0000 7fe9d0c4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:58.110 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.110+0000 7fe9c9d9b640 1 -- 172.21.15.6:0/2496754049 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9c4001b70 con 0x7fe9cc13c480 2024-03-25T19:20:58.111 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.110+0000 7fe9c9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:58.111 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.110+0000 7fe9c9d9b640 1 -- 172.21.15.6:0/2496754049 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fe9c4001f80 con 0x7fe9cc13c480 2024-03-25T19:20:58.111 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.111+0000 7fe9d0c4a640 10 monclient: get_monmap_and_config success 2024-03-25T19:20:58.111 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.111+0000 7fe9d0c4a640 4 set_mon_vals no callback set 2024-03-25T19:20:58.111 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.111+0000 7fe9c9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:58.111 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.111+0000 7fe9c9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:20:58.111 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.111+0000 7fe9c9d9b640 10 monclient: dump: 2024-03-25T19:20:58.111 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:20:58.111 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:58.111 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:58.111 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:58.111 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:58.112 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:58.112 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:20:58.112 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:20:58.112 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:20:58.112 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:58.113 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.111+0000 7fe9d0c4a640 10 monclient: shutdown 2024-03-25T19:20:58.113 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.111+0000 7fe9d0c4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:58.113 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.111+0000 7fe9d0c4a640 1 -- 172.21.15.6:0/2496754049 >> v1:172.21.15.6:6790/0 conn(0x7fe9cc13c480 legacy=0x7fe9cc14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:58.113 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.112+0000 7fe9d0c4a640 1 -- 172.21.15.6:0/2496754049 shutdown_connections 2024-03-25T19:20:58.113 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.112+0000 7fe9d0c4a640 1 -- 172.21.15.6:0/2496754049 >> 172.21.15.6:0/2496754049 conn(0x7fe9cc0617a0 msgr2=0x7fe9cc061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:58.114 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.112+0000 7fe9d0c4a640 1 -- 172.21.15.6:0/2496754049 shutdown_connections 2024-03-25T19:20:58.114 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.112+0000 7fe9d0c4a640 1 -- 172.21.15.6:0/2496754049 wait complete. 2024-03-25T19:20:58.114 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.112+0000 7fe9d0c4a640 10 monclient: build_initial_monmap 2024-03-25T19:20:58.114 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.113+0000 7fe9d0c4a640 10 monclient: monmap: 2024-03-25T19:20:58.114 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:20:58.114 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:20:58.114 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:20:58.114 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:20:58.114 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:58.114 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:20:58.114 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:20:58.114 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:20:58.114 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:20:58.114 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:58.115 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.113+0000 7fe9d0c4a640 1 librados: starting msgr at 2024-03-25T19:20:58.115 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.113+0000 7fe9d0c4a640 1 librados: starting objecter 2024-03-25T19:20:58.115 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.113+0000 7fe9d0c4a640 1 Processor -- start 2024-03-25T19:20:58.115 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.113+0000 7fe9d0c4a640 1 -- start start 2024-03-25T19:20:58.115 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.113+0000 7fe9d0c4a640 1 librados: setting wanted keys 2024-03-25T19:20:58.115 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.113+0000 7fe9d0c4a640 1 librados: calling monclient init 2024-03-25T19:20:58.115 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.113+0000 7fe9d0c4a640 10 monclient: init 2024-03-25T19:20:58.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.114+0000 7fe9d0c4a640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:58.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.114+0000 7fe9d0c4a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:20:58.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.114+0000 7fe9d0c4a640 10 monclient(hunting): picked mon.noname-a con 0x7fe9cc0b4670 addr v1:172.21.15.6:6789/0 2024-03-25T19:20:58.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.114+0000 7fe9d0c4a640 10 monclient(hunting): picked mon.noname-b con 0x7fe9cc0b4ff0 addr v1:172.21.15.192:6789/0 2024-03-25T19:20:58.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.114+0000 7fe9d0c4a640 10 monclient(hunting): picked mon.noname-c con 0x7fe9cc13c480 addr v1:172.21.15.6:6790/0 2024-03-25T19:20:58.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.114+0000 7fe9d0c4a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9cc0aeae0 con 0x7fe9cc0b4670 2024-03-25T19:20:58.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.114+0000 7fe9d0c4a640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9cc0afd70 con 0x7fe9cc0b4ff0 2024-03-25T19:20:58.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.114+0000 7fe9d0c4a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9cc0b0050 con 0x7fe9cc13c480 2024-03-25T19:20:58.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.114+0000 7fe9d0c4a640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:58.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.114+0000 7fe9d0c4a640 10 monclient(hunting): authenticate will time out at 6362.354232s 2024-03-25T19:20:58.126 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.126+0000 7fe9c3fff640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fe9cc0b4ff0 0x7fe9cc0aef70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:36026/0 (socket says 172.21.15.6:36026) 2024-03-25T19:20:58.126 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.126+0000 7fe9c3fff640 1 -- 172.21.15.6:0/2119081806 learned_addr learned my addr 172.21.15.6:0/2119081806 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:20:58.132 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.133+0000 7fe9c2ffd640 10 client.?.objecter ms_handle_connect 0x7fe9cc0b4ff0 2024-03-25T19:20:58.133 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.133+0000 7fe9c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:58.138 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.138+0000 7fe9c2ffd640 10 client.?.objecter ms_handle_connect 0x7fe9cc13c480 2024-03-25T19:20:58.138 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.138+0000 7fe9c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:58.138 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.138+0000 7fe9c2ffd640 10 client.?.objecter ms_handle_connect 0x7fe9cc0b4670 2024-03-25T19:20:58.138 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.138+0000 7fe9c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:58.141 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.141+0000 7fe9c2ffd640 1 -- 172.21.15.6:0/2119081806 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 251387114 0 0) 0x7fe9cc0afd70 con 0x7fe9cc0b4ff0 2024-03-25T19:20:58.141 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.141+0000 7fe9c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:20:58.141 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.141+0000 7fe9c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:58.141 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.141+0000 7fe9c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:58.141 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.141+0000 7fe9c2ffd640 10 monclient(hunting): my global_id is 4540 2024-03-25T19:20:58.142 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.141+0000 7fe9c2ffd640 1 -- 172.21.15.6:0/2119081806 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9ac004eb0 con 0x7fe9cc0b4ff0 2024-03-25T19:20:58.149 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.149+0000 7fe9c2ffd640 1 -- 172.21.15.6:0/2119081806 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1400848281 0 0) 0x7fe9cc0aeae0 con 0x7fe9cc0b4670 2024-03-25T19:20:58.149 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.149+0000 7fe9c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:20:58.149 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.149+0000 7fe9c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:58.149 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.149+0000 7fe9c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:58.149 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.149+0000 7fe9c2ffd640 10 monclient(hunting): my global_id is 4533 2024-03-25T19:20:58.149 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.149+0000 7fe9c2ffd640 1 -- 172.21.15.6:0/2119081806 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9cc0afd70 con 0x7fe9cc0b4670 2024-03-25T19:20:58.158 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.158+0000 7fe9c2ffd640 1 -- 172.21.15.6:0/2119081806 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 84694748 0 0) 0x7fe9ac004eb0 con 0x7fe9cc0b4ff0 2024-03-25T19:20:58.158 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.158+0000 7fe9c2ffd640 1 -- 172.21.15.6:0/2119081806 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9cc0aeae0 con 0x7fe9cc0b4ff0 2024-03-25T19:20:58.158 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.158+0000 7fe9c2ffd640 1 -- 172.21.15.6:0/2119081806 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fe9b4001a10 con 0x7fe9cc0b4ff0 2024-03-25T19:20:58.158 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.158+0000 7fe9c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:58.158 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.158+0000 7fe9c2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:20:58.158 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.158+0000 7fe9c2ffd640 10 monclient(hunting): dump: 2024-03-25T19:20:58.159 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:20:58.159 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:58.159 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:58.159 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:58.159 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:58.159 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:58.159 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:20:58.159 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:20:58.159 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:20:58.159 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:58.159 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.158+0000 7fe9c2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:20:58.160 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.160+0000 7fe9c2ffd640 1 -- 172.21.15.6:0/2119081806 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4014570565 0 0) 0x7fe9cc0b0050 con 0x7fe9cc13c480 2024-03-25T19:20:58.160 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.160+0000 7fe9c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:20:58.160 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.160+0000 7fe9c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:58.160 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.160+0000 7fe9c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:58.160 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.160+0000 7fe9c2ffd640 10 monclient(hunting): my global_id is 4535 2024-03-25T19:20:58.161 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.160+0000 7fe9c2ffd640 1 -- 172.21.15.6:0/2119081806 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9ac004eb0 con 0x7fe9cc13c480 2024-03-25T19:20:58.171 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.171+0000 7fe9c2ffd640 1 -- 172.21.15.6:0/2119081806 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2826676285 0 0) 0x7fe9cc0aeae0 con 0x7fe9cc0b4ff0 2024-03-25T19:20:58.171 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.171+0000 7fe9c2ffd640 1 -- 172.21.15.6:0/2119081806 >> v1:172.21.15.6:6790/0 conn(0x7fe9cc13c480 legacy=0x7fe9cc0adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:58.172 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.171+0000 7fe9c2ffd640 1 -- 172.21.15.6:0/2119081806 >> v1:172.21.15.6:6789/0 conn(0x7fe9cc0b4670 legacy=0x7fe9cc0b0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:58.172 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.171+0000 7fe9c2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:20:58.172 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.171+0000 7fe9c2ffd640 1 monclient: found mon.b 2024-03-25T19:20:58.172 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.171+0000 7fe9c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:58.172 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.171+0000 7fe9c2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:20:58.172 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.171+0000 7fe9c2ffd640 1 -- 172.21.15.6:0/2119081806 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9cc0b0330 con 0x7fe9cc0b4ff0 2024-03-25T19:20:58.172 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.171+0000 7fe9c2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:20:58.172 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.171+0000 7fe9c2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:20:58.172 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.171+0000 7fe9c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:58.172 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.171+0000 7fe9d0c4a640 5 monclient: authenticate success, global_id 4540 2024-03-25T19:20:58.172 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.171+0000 7fe9d0c4a640 10 monclient: _renew_subs 2024-03-25T19:20:58.172 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.171+0000 7fe9d0c4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:20:58.172 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.171+0000 7fe9d0c4a640 1 -- 172.21.15.6:0/2119081806 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9cc0b0610 con 0x7fe9cc0b4ff0 2024-03-25T19:20:58.172 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.171+0000 7fe9d0c4a640 10 client.4540.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:20:58.172 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.171+0000 7fe9d0c4a640 10 monclient: _renew_subs 2024-03-25T19:20:58.173 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.171+0000 7fe9d0c4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:20:58.173 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.171+0000 7fe9d0c4a640 1 -- 172.21.15.6:0/2119081806 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9cc1e3390 con 0x7fe9cc0b4ff0 2024-03-25T19:20:58.173 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.171+0000 7fe9d0c4a640 1 librados: init done 2024-03-25T19:20:58.173 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.172+0000 7fe9d0c4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:58.173 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.172+0000 7fe9d0c4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:58.173 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.172+0000 7fe9d0c4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:20:58.173 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.172+0000 7fe9d0c4a640 1 -- 172.21.15.6:0/2119081806 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9cc06dbd0 con 0x7fe9cc0b4ff0 2024-03-25T19:20:58.455 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.455+0000 7fe9c2ffd640 1 -- 172.21.15.6:0/2119081806 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9b4001c50 con 0x7fe9cc0b4ff0 2024-03-25T19:20:58.455 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.455+0000 7fe9c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:58.455 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.455+0000 7fe9c2ffd640 1 -- 172.21.15.6:0/2119081806 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fe9b4003a60 con 0x7fe9cc0b4ff0 2024-03-25T19:20:58.455 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.455+0000 7fe9c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:58.455 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.455+0000 7fe9c2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:20:58.455 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.455+0000 7fe9c2ffd640 10 monclient: dump: 2024-03-25T19:20:58.455 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:20:58.455 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:20:58.455 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:58.455 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:20:58.455 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:58.456 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:20:58.456 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:20:58.456 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:20:58.456 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:20:58.456 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:20:58.456 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.455+0000 7fe9c959a640 4 set_mon_vals no callback set 2024-03-25T19:20:58.456 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.456+0000 7fe9c2ffd640 1 -- 172.21.15.6:0/2119081806 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7fe9b4003c80 con 0x7fe9cc0b4ff0 2024-03-25T19:20:58.457 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.456+0000 7fe9ca59c640 10 monclient: get_auth_request con 0x7fe9ac06de20 auth_method 0 2024-03-25T19:20:58.457 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.456+0000 7fe9c2ffd640 10 client.4540.objecter ms_handle_connect 0x7fe9ac06de20 2024-03-25T19:20:58.461 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.461+0000 7fe9c2ffd640 1 -- 172.21.15.6:0/2119081806 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7fe9b4054870 con 0x7fe9cc0b4ff0 2024-03-25T19:20:58.461 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.461+0000 7fe9c2ffd640 10 client.4540.objecter ms_dispatch 0x7fe9cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:20:58.461 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.461+0000 7fe9c2ffd640 3 client.4540.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:20:58.461 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.461+0000 7fe9c2ffd640 3 client.4540.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:20:58.461 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.461+0000 7fe9c2ffd640 20 client.4540.objecter dump_active .. 0 homeless 2024-03-25T19:20:58.462 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.462+0000 7fe9c2ffd640 1 -- 172.21.15.6:0/2119081806 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe9b4050380 con 0x7fe9cc0b4ff0 2024-03-25T19:20:58.462 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.462+0000 7fe9c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:58.462 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.462+0000 7fe9c2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:20:58.687 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.687+0000 7fe9d0c4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:58.687 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.687+0000 7fe9d0c4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:58.687 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.687+0000 7fe9d0c4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:20:58.687 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.687+0000 7fe9d0c4a640 1 -- 172.21.15.6:0/2119081806 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9cc0688c0 con 0x7fe9cc0b4ff0 2024-03-25T19:20:58.719 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.719+0000 7fe9c2ffd640 1 -- 172.21.15.6:0/2119081806 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7fe9b40522b0 con 0x7fe9cc0b4ff0 2024-03-25T19:20:58.719 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.719+0000 7fe9c2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:58.719 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.719+0000 7fe9c2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:20:58.720 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:20:58.720 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:20:58.722 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.721+0000 7fe9d0c4a640 10 librados: watch_flush enter 2024-03-25T19:20:58.722 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.722+0000 7fe9d0c4a640 10 librados: watch_flush exit 2024-03-25T19:20:58.722 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.722+0000 7fe9d0c4a640 20 client.4540.objecter shutdown clearing up homeless session... 2024-03-25T19:20:58.722 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.722+0000 7fe9d0c4a640 10 client.4540.objecter successfully canceled tick 2024-03-25T19:20:58.722 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.722+0000 7fe9d0c4a640 1 -- 172.21.15.6:0/2119081806 >> v1:172.21.15.6:6800/2936652650 conn(0x7fe9ac06de20 legacy=0x7fe9ac0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:58.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.722+0000 7fe9d0c4a640 10 monclient: shutdown 2024-03-25T19:20:58.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.722+0000 7fe9d0c4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:58.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.722+0000 7fe9d0c4a640 1 -- 172.21.15.6:0/2119081806 >> v1:172.21.15.192:6789/0 conn(0x7fe9cc0b4ff0 legacy=0x7fe9cc0aef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:58.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.722+0000 7fe9d0c4a640 1 -- 172.21.15.6:0/2119081806 shutdown_connections 2024-03-25T19:20:58.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.722+0000 7fe9d0c4a640 1 -- 172.21.15.6:0/2119081806 >> 172.21.15.6:0/2119081806 conn(0x7fe9cc0617a0 msgr2=0x7fe9cc145420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:58.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.722+0000 7fe9d0c4a640 1 -- 172.21.15.6:0/2119081806 shutdown_connections 2024-03-25T19:20:58.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.722+0000 7fe9d0c4a640 1 -- 172.21.15.6:0/2119081806 wait complete. 2024-03-25T19:20:58.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:20:58.722+0000 7fe9d0c4a640 1 librados: shutdown 2024-03-25T19:20:58.734 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:21:05.539 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:21:05.652 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.652+0000 7f278dd40640 10 monclient: get_monmap_and_config 2024-03-25T19:21:05.652 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.652+0000 7f278dd40640 10 monclient: build_initial_monmap 2024-03-25T19:21:05.652 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.652+0000 7f278dd40640 10 monclient: monmap: 2024-03-25T19:21:05.652 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:21:05.652 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:05.652 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:21:05.653342+0000 2024-03-25T19:21:05.653 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:21:05.653342+0000 2024-03-25T19:21:05.653 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:05.653 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:05.653 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:21:05.653 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:21:05.653 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:21:05.653 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:05.654 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.653+0000 7f278dd40640 1 Processor -- start 2024-03-25T19:21:05.654 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.653+0000 7f278dd40640 1 -- start start 2024-03-25T19:21:05.654 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.653+0000 7f278dd40640 10 monclient: init 2024-03-25T19:21:05.654 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.653+0000 7f278dd40640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:05.654 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.654+0000 7f278dd40640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:21:05.654 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.654+0000 7f278dd40640 10 monclient(hunting): picked mon.noname-b con 0x7f27880b4ff0 addr v1:172.21.15.6:6790/0 2024-03-25T19:21:05.654 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.654+0000 7f278dd40640 10 monclient(hunting): picked mon.noname-c con 0x7f27880b4670 addr v1:172.21.15.192:6789/0 2024-03-25T19:21:05.654 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.654+0000 7f278dd40640 10 monclient(hunting): picked mon.noname-a con 0x7f278813c480 addr v1:172.21.15.6:6789/0 2024-03-25T19:21:05.655 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.654+0000 7f278dd40640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27880ab0e0 con 0x7f278813c480 2024-03-25T19:21:05.655 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.654+0000 7f278dd40640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27880ab3c0 con 0x7f27880b4670 2024-03-25T19:21:05.655 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.654+0000 7f278dd40640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27880a9f30 con 0x7f27880b4ff0 2024-03-25T19:21:05.655 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.654+0000 7f278dd40640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:05.655 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.654+0000 7f278dd40640 10 monclient(hunting): authenticate will time out at 6369.894016s 2024-03-25T19:21:05.672 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.672+0000 7f2786ffd640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f27880b4670 0x7f27881425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:47484/0 (socket says 172.21.15.6:47484) 2024-03-25T19:21:05.672 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.672+0000 7f2786ffd640 1 -- 172.21.15.6:0/370577346 learned_addr learned my addr 172.21.15.6:0/370577346 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:21:05.687 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.687+0000 7f27867fc640 1 -- 172.21.15.6:0/370577346 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1991443150 0 0) 0x7f27880ab3c0 con 0x7f27880b4670 2024-03-25T19:21:05.687 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.687+0000 7f27867fc640 10 monclient(hunting): _negotiate 2024-03-25T19:21:05.687 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.687+0000 7f27867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:05.687 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.687+0000 7f27867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:05.687 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.687+0000 7f27867fc640 10 monclient(hunting): my global_id is 4543 2024-03-25T19:21:05.687 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.687+0000 7f27867fc640 1 -- 172.21.15.6:0/370577346 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2770003520 con 0x7f27880b4670 2024-03-25T19:21:05.704 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.703+0000 7f27867fc640 1 -- 172.21.15.6:0/370577346 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2922407350 0 0) 0x7f27880ab0e0 con 0x7f278813c480 2024-03-25T19:21:05.704 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.704+0000 7f27867fc640 10 monclient(hunting): _negotiate 2024-03-25T19:21:05.704 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.704+0000 7f27867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:05.704 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.704+0000 7f27867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:05.704 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.704+0000 7f27867fc640 10 monclient(hunting): my global_id is 4536 2024-03-25T19:21:05.704 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.704+0000 7f27867fc640 1 -- 172.21.15.6:0/370577346 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27880ab3c0 con 0x7f278813c480 2024-03-25T19:21:05.704 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.704+0000 7f27867fc640 1 -- 172.21.15.6:0/370577346 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 180212606 0 0) 0x7f2770003520 con 0x7f27880b4670 2024-03-25T19:21:05.704 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.704+0000 7f27867fc640 1 -- 172.21.15.6:0/370577346 >> v1:172.21.15.6:6790/0 conn(0x7f27880b4ff0 legacy=0x7f278813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:05.705 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.704+0000 7f27867fc640 1 -- 172.21.15.6:0/370577346 >> v1:172.21.15.6:6789/0 conn(0x7f278813c480 legacy=0x7f278814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:05.705 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.704+0000 7f27867fc640 10 monclient: _finish_hunting 0 2024-03-25T19:21:05.705 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.704+0000 7f27867fc640 1 monclient: found mon.noname-c 2024-03-25T19:21:05.705 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.704+0000 7f27867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:05.705 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.704+0000 7f27867fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:21:05.705 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.704+0000 7f27867fc640 1 -- 172.21.15.6:0/370577346 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27880a9b70 con 0x7f27880b4670 2024-03-25T19:21:05.705 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.704+0000 7f27867fc640 10 monclient: _finish_auth 0 2024-03-25T19:21:05.705 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.704+0000 7f27867fc640 10 monclient: _check_auth_tickets 2024-03-25T19:21:05.705 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.704+0000 7f27867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:05.705 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.704+0000 7f27867fc640 1 -- 172.21.15.6:0/370577346 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f277c001690 con 0x7f27880b4670 2024-03-25T19:21:05.705 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.704+0000 7f27867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:05.705 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.704+0000 7f27867fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:21:05.705 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.704+0000 7f27867fc640 10 monclient: dump: 2024-03-25T19:21:05.705 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:21:05.705 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:05.705 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:05.706 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:05.706 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:05.706 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:05.706 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:21:05.706 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:21:05.706 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:21:05.706 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:05.706 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.704+0000 7f278dd40640 5 monclient: authenticate success, global_id 4543 2024-03-25T19:21:05.706 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.704+0000 7f278dd40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:21:05.733 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.733+0000 7f27867fc640 1 -- 172.21.15.6:0/370577346 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f277c0017f0 con 0x7f27880b4670 2024-03-25T19:21:05.734 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.734+0000 7f27867fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:05.734 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.734+0000 7f27867fc640 1 -- 172.21.15.6:0/370577346 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f277c001c00 con 0x7f27880b4670 2024-03-25T19:21:05.734 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.734+0000 7f278dd40640 10 monclient: get_monmap_and_config success 2024-03-25T19:21:05.734 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.734+0000 7f278dd40640 4 set_mon_vals no callback set 2024-03-25T19:21:05.734 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.734+0000 7f27867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:05.734 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.734+0000 7f27867fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:21:05.734 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.734+0000 7f27867fc640 10 monclient: dump: 2024-03-25T19:21:05.734 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:21:05.734 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:05.734 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:05.734 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:05.734 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:05.734 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:05.735 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:21:05.735 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:21:05.735 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:21:05.735 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:05.736 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.734+0000 7f278dd40640 10 monclient: shutdown 2024-03-25T19:21:05.736 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.734+0000 7f278dd40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:05.736 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.734+0000 7f278dd40640 1 -- 172.21.15.6:0/370577346 >> v1:172.21.15.192:6789/0 conn(0x7f27880b4670 legacy=0x7f27881425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:05.736 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.734+0000 7f278dd40640 1 -- 172.21.15.6:0/370577346 shutdown_connections 2024-03-25T19:21:05.736 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.734+0000 7f278dd40640 1 -- 172.21.15.6:0/370577346 >> 172.21.15.6:0/370577346 conn(0x7f27880617a0 msgr2=0x7f2788061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:05.737 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.735+0000 7f278dd40640 1 -- 172.21.15.6:0/370577346 shutdown_connections 2024-03-25T19:21:05.737 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.735+0000 7f278dd40640 1 -- 172.21.15.6:0/370577346 wait complete. 2024-03-25T19:21:05.737 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.735+0000 7f278dd40640 10 monclient: build_initial_monmap 2024-03-25T19:21:05.737 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.736+0000 7f278dd40640 10 monclient: monmap: 2024-03-25T19:21:05.737 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:21:05.737 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:21:05.737 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:21:05.737 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:21:05.737 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:05.737 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:21:05.737 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:21:05.737 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:21:05.737 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:21:05.737 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:05.737 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.736+0000 7f278dd40640 1 librados: starting msgr at 2024-03-25T19:21:05.737 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.736+0000 7f278dd40640 1 librados: starting objecter 2024-03-25T19:21:05.738 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.736+0000 7f278dd40640 1 Processor -- start 2024-03-25T19:21:05.738 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.736+0000 7f278dd40640 1 -- start start 2024-03-25T19:21:05.738 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.736+0000 7f278dd40640 1 librados: setting wanted keys 2024-03-25T19:21:05.738 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.736+0000 7f278dd40640 1 librados: calling monclient init 2024-03-25T19:21:05.738 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.736+0000 7f278dd40640 10 monclient: init 2024-03-25T19:21:05.739 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.737+0000 7f278dd40640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:05.739 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.737+0000 7f278dd40640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:21:05.739 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.737+0000 7f278dd40640 10 monclient(hunting): picked mon.noname-c con 0x7f27880b4670 addr v1:172.21.15.6:6790/0 2024-03-25T19:21:05.739 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.737+0000 7f278dd40640 10 monclient(hunting): picked mon.noname-a con 0x7f27880b4ff0 addr v1:172.21.15.6:6789/0 2024-03-25T19:21:05.739 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.737+0000 7f278dd40640 10 monclient(hunting): picked mon.noname-b con 0x7f278813c480 addr v1:172.21.15.192:6789/0 2024-03-25T19:21:05.739 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.737+0000 7f278dd40640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2788143810 con 0x7f27880b4ff0 2024-03-25T19:21:05.739 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.737+0000 7f278dd40640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2788130ad0 con 0x7f278813c480 2024-03-25T19:21:05.739 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.737+0000 7f278dd40640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2788144120 con 0x7f27880b4670 2024-03-25T19:21:05.739 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.737+0000 7f278dd40640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:05.739 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.737+0000 7f278dd40640 10 monclient(hunting): authenticate will time out at 6369.977223s 2024-03-25T19:21:05.744 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.745+0000 7f2786ffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f27880b4ff0 0x7f278812fcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46354/0 (socket says 172.21.15.6:46354) 2024-03-25T19:21:05.745 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.745+0000 7f2786ffd640 1 -- 172.21.15.6:0/2982593613 learned_addr learned my addr 172.21.15.6:0/2982593613 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:21:05.745 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.745+0000 7f276ffff640 10 client.?.objecter ms_handle_connect 0x7f27880b4ff0 2024-03-25T19:21:05.745 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.745+0000 7f276ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:05.750 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.750+0000 7f276ffff640 10 client.?.objecter ms_handle_connect 0x7f27880b4670 2024-03-25T19:21:05.750 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.750+0000 7f276ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:05.751 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.751+0000 7f276ffff640 10 client.?.objecter ms_handle_connect 0x7f278813c480 2024-03-25T19:21:05.751 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.751+0000 7f276ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:05.761 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.761+0000 7f276ffff640 1 -- 172.21.15.6:0/2982593613 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1473634549 0 0) 0x7f2788143810 con 0x7f27880b4ff0 2024-03-25T19:21:05.761 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.761+0000 7f276ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:21:05.761 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.761+0000 7f276ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:05.761 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.761+0000 7f276ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:05.761 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.761+0000 7f276ffff640 10 monclient(hunting): my global_id is 4539 2024-03-25T19:21:05.761 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.761+0000 7f276ffff640 1 -- 172.21.15.6:0/2982593613 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2774004eb0 con 0x7f27880b4ff0 2024-03-25T19:21:05.768 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.768+0000 7f276ffff640 1 -- 172.21.15.6:0/2982593613 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1266999289 0 0) 0x7f2788130ad0 con 0x7f278813c480 2024-03-25T19:21:05.768 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.768+0000 7f276ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:21:05.769 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.768+0000 7f276ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:05.769 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.768+0000 7f276ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:05.769 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.768+0000 7f276ffff640 10 monclient(hunting): my global_id is 4546 2024-03-25T19:21:05.769 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.768+0000 7f276ffff640 1 -- 172.21.15.6:0/2982593613 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2788143810 con 0x7f278813c480 2024-03-25T19:21:05.770 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.770+0000 7f276ffff640 1 -- 172.21.15.6:0/2982593613 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2379268047 0 0) 0x7f2788144120 con 0x7f27880b4670 2024-03-25T19:21:05.770 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.770+0000 7f276ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:21:05.770 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.770+0000 7f276ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:05.770 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.770+0000 7f276ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:05.770 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.770+0000 7f276ffff640 10 monclient(hunting): my global_id is 4541 2024-03-25T19:21:05.770 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.770+0000 7f276ffff640 1 -- 172.21.15.6:0/2982593613 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2788130ad0 con 0x7f27880b4670 2024-03-25T19:21:05.779 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.779+0000 7f276ffff640 1 -- 172.21.15.6:0/2982593613 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 746032552 0 0) 0x7f2788143810 con 0x7f278813c480 2024-03-25T19:21:05.779 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.779+0000 7f276ffff640 1 -- 172.21.15.6:0/2982593613 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2788144120 con 0x7f278813c480 2024-03-25T19:21:05.779 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.779+0000 7f276ffff640 1 -- 172.21.15.6:0/2982593613 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f2780001b90 con 0x7f278813c480 2024-03-25T19:21:05.780 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.779+0000 7f276ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:05.780 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.779+0000 7f276ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:21:05.780 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.779+0000 7f276ffff640 10 monclient(hunting): dump: 2024-03-25T19:21:05.780 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:21:05.780 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:05.780 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:05.780 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:05.780 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:05.780 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:05.780 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:21:05.780 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:21:05.780 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:21:05.780 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:05.780 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.779+0000 7f276ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:21:05.785 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.785+0000 7f276ffff640 1 -- 172.21.15.6:0/2982593613 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2058698072 0 0) 0x7f2774004eb0 con 0x7f27880b4ff0 2024-03-25T19:21:05.785 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.785+0000 7f276ffff640 1 -- 172.21.15.6:0/2982593613 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2788143810 con 0x7f27880b4ff0 2024-03-25T19:21:05.785 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.785+0000 7f276ffff640 1 -- 172.21.15.6:0/2982593613 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f277c001c00 con 0x7f27880b4ff0 2024-03-25T19:21:05.785 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.785+0000 7f276ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:05.785 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.785+0000 7f276ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:21:05.785 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.785+0000 7f276ffff640 10 monclient(hunting): dump: 2024-03-25T19:21:05.785 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:21:05.786 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:05.786 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:05.786 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:05.786 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:05.786 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:05.786 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:21:05.786 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:21:05.786 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:21:05.786 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:05.790 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.790+0000 7f276ffff640 1 -- 172.21.15.6:0/2982593613 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4171363556 0 0) 0x7f2788130ad0 con 0x7f27880b4670 2024-03-25T19:21:05.790 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.790+0000 7f276ffff640 1 -- 172.21.15.6:0/2982593613 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2774004eb0 con 0x7f27880b4670 2024-03-25T19:21:05.790 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.790+0000 7f276ffff640 1 -- 172.21.15.6:0/2982593613 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f2778001860 con 0x7f27880b4670 2024-03-25T19:21:05.790 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.790+0000 7f276ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:05.790 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.790+0000 7f276ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:21:05.790 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.790+0000 7f276ffff640 10 monclient(hunting): dump: 2024-03-25T19:21:05.790 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:21:05.790 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:05.791 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:05.791 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:05.791 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:05.791 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:05.791 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:21:05.791 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:21:05.791 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:21:05.791 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:05.794 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.794+0000 7f276ffff640 1 -- 172.21.15.6:0/2982593613 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3765996944 0 0) 0x7f2788144120 con 0x7f278813c480 2024-03-25T19:21:05.794 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.794+0000 7f276ffff640 1 -- 172.21.15.6:0/2982593613 >> v1:172.21.15.6:6790/0 conn(0x7f27880b4670 legacy=0x7f2788144de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:05.794 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.794+0000 7f276ffff640 1 -- 172.21.15.6:0/2982593613 >> v1:172.21.15.6:6789/0 conn(0x7f27880b4ff0 legacy=0x7f278812fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:05.795 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.794+0000 7f276ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:21:05.795 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.794+0000 7f276ffff640 1 monclient: found mon.b 2024-03-25T19:21:05.795 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.794+0000 7f276ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:05.795 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.794+0000 7f276ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:21:05.795 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.794+0000 7f276ffff640 1 -- 172.21.15.6:0/2982593613 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2788144400 con 0x7f278813c480 2024-03-25T19:21:05.795 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.794+0000 7f276ffff640 10 monclient: _finish_auth 0 2024-03-25T19:21:05.795 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.794+0000 7f276ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:21:05.795 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.794+0000 7f276ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:05.795 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.794+0000 7f278dd40640 5 monclient: authenticate success, global_id 4546 2024-03-25T19:21:05.795 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.794+0000 7f278dd40640 10 monclient: _renew_subs 2024-03-25T19:21:05.795 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.794+0000 7f278dd40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:21:05.795 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.794+0000 7f278dd40640 1 -- 172.21.15.6:0/2982593613 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27881446e0 con 0x7f278813c480 2024-03-25T19:21:05.795 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.794+0000 7f278dd40640 10 client.4546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:21:05.795 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.794+0000 7f278dd40640 10 monclient: _renew_subs 2024-03-25T19:21:05.795 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.794+0000 7f278dd40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:21:05.796 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.794+0000 7f278dd40640 1 -- 172.21.15.6:0/2982593613 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f27881454f0 con 0x7f278813c480 2024-03-25T19:21:05.796 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.794+0000 7f278dd40640 1 librados: init done 2024-03-25T19:21:05.796 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.795+0000 7f278dd40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:05.796 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.795+0000 7f278dd40640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:05.796 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.795+0000 7f278dd40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:21:05.796 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:05.795+0000 7f278dd40640 1 -- 172.21.15.6:0/2982593613 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f278806dbd0 con 0x7f278813c480 2024-03-25T19:21:06.064 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:06.064+0000 7f276ffff640 1 -- 172.21.15.6:0/2982593613 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2780002040 con 0x7f278813c480 2024-03-25T19:21:06.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:06.064+0000 7f276ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:06.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:06.064+0000 7f276ffff640 1 -- 172.21.15.6:0/2982593613 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f2780003b10 con 0x7f278813c480 2024-03-25T19:21:06.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:06.064+0000 7f276ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:06.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:06.064+0000 7f276ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:21:06.065 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:06.064+0000 7f276ffff640 10 monclient: dump: 2024-03-25T19:21:06.065 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:21:06.065 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:06.065 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:06.065 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:06.065 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:06.065 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:06.065 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:21:06.065 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:21:06.065 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:21:06.066 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:06.066 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:06.064+0000 7f2785ffb640 4 set_mon_vals no callback set 2024-03-25T19:21:06.066 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:06.065+0000 7f276ffff640 1 -- 172.21.15.6:0/2982593613 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f2780019ae0 con 0x7f278813c480 2024-03-25T19:21:06.066 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:06.066+0000 7f27877fe640 10 monclient: get_auth_request con 0x7f277406e0b0 auth_method 0 2024-03-25T19:21:06.066 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:06.066+0000 7f276ffff640 10 client.4546.objecter ms_handle_connect 0x7f277406e0b0 2024-03-25T19:21:06.070 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:06.070+0000 7f276ffff640 1 -- 172.21.15.6:0/2982593613 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f2780053820 con 0x7f278813c480 2024-03-25T19:21:06.071 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:06.070+0000 7f276ffff640 10 client.4546.objecter ms_dispatch 0x7f2788000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:21:06.071 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:06.070+0000 7f276ffff640 3 client.4546.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:21:06.071 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:06.070+0000 7f276ffff640 3 client.4546.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:21:06.071 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:06.070+0000 7f276ffff640 20 client.4546.objecter dump_active .. 0 homeless 2024-03-25T19:21:06.071 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:06.070+0000 7f276ffff640 1 -- 172.21.15.6:0/2982593613 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f278004f160 con 0x7f278813c480 2024-03-25T19:21:06.071 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:06.070+0000 7f276ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:06.071 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:06.070+0000 7f276ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:21:06.289 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:06.289+0000 7f278dd40640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:06.290 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:06.289+0000 7f278dd40640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:06.290 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:06.289+0000 7f278dd40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:21:06.290 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:06.289+0000 7f278dd40640 1 -- 172.21.15.6:0/2982593613 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27880688c0 con 0x7f278813c480 2024-03-25T19:21:06.314 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:06.314+0000 7f276ffff640 1 -- 172.21.15.6:0/2982593613 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f2780051260 con 0x7f278813c480 2024-03-25T19:21:06.314 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:06.314+0000 7f276ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:06.314 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:06.314+0000 7f276ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:21:06.315 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:21:06.315 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:21:06.318 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:06.316+0000 7f278dd40640 10 librados: watch_flush enter 2024-03-25T19:21:06.318 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:06.316+0000 7f278dd40640 10 librados: watch_flush exit 2024-03-25T19:21:06.319 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:06.316+0000 7f278dd40640 20 client.4546.objecter shutdown clearing up homeless session... 2024-03-25T19:21:06.319 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:06.316+0000 7f278dd40640 10 client.4546.objecter successfully canceled tick 2024-03-25T19:21:06.319 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:06.316+0000 7f278dd40640 1 -- 172.21.15.6:0/2982593613 >> v1:172.21.15.6:6800/2936652650 conn(0x7f277406e0b0 legacy=0x7f2774070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:06.319 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:06.316+0000 7f278dd40640 10 monclient: shutdown 2024-03-25T19:21:06.319 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:06.316+0000 7f278dd40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:06.319 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:06.316+0000 7f278dd40640 1 -- 172.21.15.6:0/2982593613 >> v1:172.21.15.192:6789/0 conn(0x7f278813c480 legacy=0x7f27881303c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:06.319 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:06.317+0000 7f278dd40640 1 -- 172.21.15.6:0/2982593613 shutdown_connections 2024-03-25T19:21:06.319 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:06.317+0000 7f278dd40640 1 -- 172.21.15.6:0/2982593613 >> 172.21.15.6:0/2982593613 conn(0x7f27880617a0 msgr2=0x7f2788140760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:06.319 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:06.317+0000 7f278dd40640 1 -- 172.21.15.6:0/2982593613 shutdown_connections 2024-03-25T19:21:06.319 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:06.317+0000 7f278dd40640 1 -- 172.21.15.6:0/2982593613 wait complete. 2024-03-25T19:21:06.319 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:06.317+0000 7f278dd40640 1 librados: shutdown 2024-03-25T19:21:06.328 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:21:08.725 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:07:26.072 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:08.775 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:07:26.093 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:08.777 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:07:26.124 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:08.778 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:07:26.095 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:08.803 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:07:26.149 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:08.816 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:07:26.134 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:08.861 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:07:26.207 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:08.868 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:07:26.215 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:08.903 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:07:26.250 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:13.134 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:21:13.252 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.252+0000 7f0a436d6640 10 monclient: get_monmap_and_config 2024-03-25T19:21:13.252 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.252+0000 7f0a436d6640 10 monclient: build_initial_monmap 2024-03-25T19:21:13.252 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.252+0000 7f0a436d6640 10 monclient: monmap: 2024-03-25T19:21:13.252 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:21:13.252 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:13.252 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:21:13.253182+0000 2024-03-25T19:21:13.252 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:21:13.253182+0000 2024-03-25T19:21:13.252 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:13.253 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:13.253 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:21:13.253 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:21:13.253 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:21:13.253 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:13.254 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.252+0000 7f0a436d6640 1 Processor -- start 2024-03-25T19:21:13.254 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.253+0000 7f0a436d6640 1 -- start start 2024-03-25T19:21:13.254 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.253+0000 7f0a436d6640 10 monclient: init 2024-03-25T19:21:13.254 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.253+0000 7f0a436d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:13.254 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.253+0000 7f0a436d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:21:13.255 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.253+0000 7f0a436d6640 10 monclient(hunting): picked mon.noname-a con 0x7f0a3c0b1a70 addr v1:172.21.15.6:6789/0 2024-03-25T19:21:13.255 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.253+0000 7f0a436d6640 10 monclient(hunting): picked mon.noname-b con 0x7f0a3c146490 addr v1:172.21.15.6:6790/0 2024-03-25T19:21:13.255 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.254+0000 7f0a436d6640 10 monclient(hunting): picked mon.noname-c con 0x7f0a3c1458a0 addr v1:172.21.15.192:6789/0 2024-03-25T19:21:13.255 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.254+0000 7f0a436d6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a3c145d30 con 0x7f0a3c0b1a70 2024-03-25T19:21:13.255 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.254+0000 7f0a436d6640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a3c13ffb0 con 0x7f0a3c1458a0 2024-03-25T19:21:13.255 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.254+0000 7f0a436d6640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a3c0b26a0 con 0x7f0a3c146490 2024-03-25T19:21:13.255 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.254+0000 7f0a436d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:13.255 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.254+0000 7f0a436d6640 10 monclient(hunting): authenticate will time out at 6377.493843s 2024-03-25T19:21:13.259 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.259+0000 7f0a41c73640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f0a3c1458a0 0x7f0a3c14dc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:32962/0 (socket says 172.21.15.6:32962) 2024-03-25T19:21:13.259 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.259+0000 7f0a41c73640 1 -- 172.21.15.6:0/763081397 learned_addr learned my addr 172.21.15.6:0/763081397 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:21:13.282 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.282+0000 7f0a33fff640 1 -- 172.21.15.6:0/763081397 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2526045036 0 0) 0x7f0a3c0b26a0 con 0x7f0a3c146490 2024-03-25T19:21:13.282 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.282+0000 7f0a33fff640 10 monclient(hunting): _negotiate 2024-03-25T19:21:13.282 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.282+0000 7f0a33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:13.282 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.282+0000 7f0a33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:13.282 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.282+0000 7f0a33fff640 10 monclient(hunting): my global_id is 4544 2024-03-25T19:21:13.282 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.282+0000 7f0a33fff640 1 -- 172.21.15.6:0/763081397 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a28003520 con 0x7f0a3c146490 2024-03-25T19:21:13.285 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.285+0000 7f0a33fff640 1 -- 172.21.15.6:0/763081397 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1516201823 0 0) 0x7f0a3c13ffb0 con 0x7f0a3c1458a0 2024-03-25T19:21:13.285 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.285+0000 7f0a33fff640 10 monclient(hunting): _negotiate 2024-03-25T19:21:13.285 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.285+0000 7f0a33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:13.285 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.285+0000 7f0a33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:13.285 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.285+0000 7f0a33fff640 10 monclient(hunting): my global_id is 4549 2024-03-25T19:21:13.285 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.285+0000 7f0a33fff640 1 -- 172.21.15.6:0/763081397 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a3c0b26a0 con 0x7f0a3c1458a0 2024-03-25T19:21:13.290 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.289+0000 7f0a33fff640 1 -- 172.21.15.6:0/763081397 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 75887750 0 0) 0x7f0a28003520 con 0x7f0a3c146490 2024-03-25T19:21:13.290 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.289+0000 7f0a33fff640 1 -- 172.21.15.6:0/763081397 >> v1:172.21.15.192:6789/0 conn(0x7f0a3c1458a0 legacy=0x7f0a3c14dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:13.290 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.289+0000 7f0a33fff640 1 -- 172.21.15.6:0/763081397 >> v1:172.21.15.6:6789/0 conn(0x7f0a3c0b1a70 legacy=0x7f0a3c06bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:13.290 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.289+0000 7f0a33fff640 10 monclient: _finish_hunting 0 2024-03-25T19:21:13.290 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.289+0000 7f0a33fff640 1 monclient: found mon.noname-b 2024-03-25T19:21:13.290 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.289+0000 7f0a33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:13.290 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.289+0000 7f0a33fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-03-25T19:21:13.290 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.289+0000 7f0a33fff640 1 -- 172.21.15.6:0/763081397 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a3c0ae310 con 0x7f0a3c146490 2024-03-25T19:21:13.290 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.289+0000 7f0a33fff640 10 monclient: _finish_auth 0 2024-03-25T19:21:13.291 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.289+0000 7f0a33fff640 10 monclient: _check_auth_tickets 2024-03-25T19:21:13.291 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.289+0000 7f0a33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:13.291 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.289+0000 7f0a33fff640 1 -- 172.21.15.6:0/763081397 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f0a2c001690 con 0x7f0a3c146490 2024-03-25T19:21:13.291 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.289+0000 7f0a33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:13.291 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.289+0000 7f0a33fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:21:13.291 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.290+0000 7f0a33fff640 10 monclient: dump: 2024-03-25T19:21:13.291 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:21:13.291 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:13.291 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:13.291 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:13.291 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:13.291 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:13.291 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:21:13.291 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:21:13.291 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:21:13.292 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:13.292 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.290+0000 7f0a436d6640 5 monclient: authenticate success, global_id 4544 2024-03-25T19:21:13.292 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.290+0000 7f0a436d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:21:13.302 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.302+0000 7f0a33fff640 1 -- 172.21.15.6:0/763081397 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a2c0017f0 con 0x7f0a3c146490 2024-03-25T19:21:13.302 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.302+0000 7f0a33fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:13.302 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.302+0000 7f0a33fff640 1 -- 172.21.15.6:0/763081397 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f0a2c001c00 con 0x7f0a3c146490 2024-03-25T19:21:13.302 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.302+0000 7f0a436d6640 10 monclient: get_monmap_and_config success 2024-03-25T19:21:13.302 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.302+0000 7f0a436d6640 4 set_mon_vals no callback set 2024-03-25T19:21:13.302 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.302+0000 7f0a33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:13.302 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.302+0000 7f0a33fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:21:13.302 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.302+0000 7f0a33fff640 10 monclient: dump: 2024-03-25T19:21:13.302 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:21:13.302 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:13.303 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:13.303 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:13.303 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:13.303 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:13.303 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:21:13.303 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:21:13.303 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:21:13.303 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:13.304 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.302+0000 7f0a436d6640 10 monclient: shutdown 2024-03-25T19:21:13.305 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.302+0000 7f0a436d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:13.305 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.302+0000 7f0a436d6640 1 -- 172.21.15.6:0/763081397 >> v1:172.21.15.6:6790/0 conn(0x7f0a3c146490 legacy=0x7f0a3c0afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:13.305 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.302+0000 7f0a436d6640 1 -- 172.21.15.6:0/763081397 shutdown_connections 2024-03-25T19:21:13.305 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.303+0000 7f0a436d6640 1 -- 172.21.15.6:0/763081397 >> 172.21.15.6:0/763081397 conn(0x7f0a3c0b28f0 msgr2=0x7f0a3c0b2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:13.305 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.303+0000 7f0a436d6640 1 -- 172.21.15.6:0/763081397 shutdown_connections 2024-03-25T19:21:13.305 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.303+0000 7f0a436d6640 1 -- 172.21.15.6:0/763081397 wait complete. 2024-03-25T19:21:13.305 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.303+0000 7f0a436d6640 10 monclient: build_initial_monmap 2024-03-25T19:21:13.305 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.304+0000 7f0a436d6640 10 monclient: monmap: 2024-03-25T19:21:13.305 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:21:13.305 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:21:13.305 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:21:13.305 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:21:13.305 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:13.305 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:21:13.306 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:21:13.306 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:21:13.306 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:21:13.306 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:13.306 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.304+0000 7f0a436d6640 1 librados: starting msgr at 2024-03-25T19:21:13.306 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.304+0000 7f0a436d6640 1 librados: starting objecter 2024-03-25T19:21:13.306 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.304+0000 7f0a436d6640 1 Processor -- start 2024-03-25T19:21:13.306 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.304+0000 7f0a436d6640 1 -- start start 2024-03-25T19:21:13.306 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.304+0000 7f0a436d6640 1 librados: setting wanted keys 2024-03-25T19:21:13.306 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.304+0000 7f0a436d6640 1 librados: calling monclient init 2024-03-25T19:21:13.306 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.304+0000 7f0a436d6640 10 monclient: init 2024-03-25T19:21:13.307 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.305+0000 7f0a436d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:13.307 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.305+0000 7f0a436d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:21:13.307 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.305+0000 7f0a436d6640 10 monclient(hunting): picked mon.noname-a con 0x7f0a3c0b1a70 addr v1:172.21.15.6:6789/0 2024-03-25T19:21:13.307 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.305+0000 7f0a436d6640 10 monclient(hunting): picked mon.noname-c con 0x7f0a3c1458a0 addr v1:172.21.15.6:6790/0 2024-03-25T19:21:13.308 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.305+0000 7f0a436d6640 10 monclient(hunting): picked mon.noname-b con 0x7f0a3c146490 addr v1:172.21.15.192:6789/0 2024-03-25T19:21:13.308 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.306+0000 7f0a436d6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a3c142880 con 0x7f0a3c0b1a70 2024-03-25T19:21:13.308 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.306+0000 7f0a436d6640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a3c1448b0 con 0x7f0a3c146490 2024-03-25T19:21:13.308 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.306+0000 7f0a436d6640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a3c144a40 con 0x7f0a3c1458a0 2024-03-25T19:21:13.308 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.306+0000 7f0a436d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:13.308 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.306+0000 7f0a436d6640 10 monclient(hunting): authenticate will time out at 6377.546128s 2024-03-25T19:21:13.316 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.316+0000 7f0a40c71640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f0a3c1458a0 0x7f0a3c1daa30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45096/0 (socket says 172.21.15.6:45096) 2024-03-25T19:21:13.316 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.316+0000 7f0a40c71640 1 -- 172.21.15.6:0/1577915763 learned_addr learned my addr 172.21.15.6:0/1577915763 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:21:13.320 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.320+0000 7f0a31ffb640 10 client.?.objecter ms_handle_connect 0x7f0a3c1458a0 2024-03-25T19:21:13.320 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.320+0000 7f0a31ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:13.327 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.327+0000 7f0a31ffb640 10 client.?.objecter ms_handle_connect 0x7f0a3c146490 2024-03-25T19:21:13.327 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.327+0000 7f0a31ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:13.331 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.331+0000 7f0a31ffb640 10 client.?.objecter ms_handle_connect 0x7f0a3c0b1a70 2024-03-25T19:21:13.331 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.331+0000 7f0a31ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:13.335 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.335+0000 7f0a31ffb640 1 -- 172.21.15.6:0/1577915763 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4208378372 0 0) 0x7f0a3c144a40 con 0x7f0a3c1458a0 2024-03-25T19:21:13.335 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.335+0000 7f0a31ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:21:13.335 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.335+0000 7f0a31ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:13.335 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.335+0000 7f0a31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:13.335 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.335+0000 7f0a31ffb640 10 monclient(hunting): my global_id is 4547 2024-03-25T19:21:13.336 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.335+0000 7f0a31ffb640 1 -- 172.21.15.6:0/1577915763 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a20004eb0 con 0x7f0a3c1458a0 2024-03-25T19:21:13.343 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.343+0000 7f0a31ffb640 1 -- 172.21.15.6:0/1577915763 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3185859590 0 0) 0x7f0a3c142880 con 0x7f0a3c0b1a70 2024-03-25T19:21:13.343 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.343+0000 7f0a31ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:21:13.343 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.343+0000 7f0a31ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:13.343 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.343+0000 7f0a31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:13.343 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.343+0000 7f0a31ffb640 10 monclient(hunting): my global_id is 4545 2024-03-25T19:21:13.343 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.343+0000 7f0a31ffb640 1 -- 172.21.15.6:0/1577915763 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a3c144a40 con 0x7f0a3c0b1a70 2024-03-25T19:21:13.346 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.346+0000 7f0a31ffb640 1 -- 172.21.15.6:0/1577915763 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1404384422 0 0) 0x7f0a3c1448b0 con 0x7f0a3c146490 2024-03-25T19:21:13.346 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.346+0000 7f0a31ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:21:13.346 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.346+0000 7f0a31ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:13.347 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.346+0000 7f0a31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:13.347 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.346+0000 7f0a31ffb640 10 monclient(hunting): my global_id is 4552 2024-03-25T19:21:13.347 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.346+0000 7f0a31ffb640 1 -- 172.21.15.6:0/1577915763 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a3c142880 con 0x7f0a3c146490 2024-03-25T19:21:13.351 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.351+0000 7f0a31ffb640 1 -- 172.21.15.6:0/1577915763 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3582177675 0 0) 0x7f0a20004eb0 con 0x7f0a3c1458a0 2024-03-25T19:21:13.351 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.351+0000 7f0a31ffb640 1 -- 172.21.15.6:0/1577915763 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a3c1448b0 con 0x7f0a3c1458a0 2024-03-25T19:21:13.351 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.351+0000 7f0a31ffb640 1 -- 172.21.15.6:0/1577915763 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f0a2c001c40 con 0x7f0a3c1458a0 2024-03-25T19:21:13.351 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.351+0000 7f0a31ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:13.351 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.351+0000 7f0a31ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:21:13.352 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.351+0000 7f0a31ffb640 10 monclient(hunting): dump: 2024-03-25T19:21:13.352 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:21:13.352 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:13.352 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:13.352 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:13.352 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:13.352 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:13.352 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:21:13.352 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:21:13.352 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:21:13.352 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:13.352 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.351+0000 7f0a31ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:21:13.361 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.361+0000 7f0a31ffb640 1 -- 172.21.15.6:0/1577915763 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 42971302 0 0) 0x7f0a3c1448b0 con 0x7f0a3c1458a0 2024-03-25T19:21:13.361 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.361+0000 7f0a31ffb640 1 -- 172.21.15.6:0/1577915763 >> v1:172.21.15.192:6789/0 conn(0x7f0a3c146490 legacy=0x7f0a3c1e1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:13.361 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.361+0000 7f0a31ffb640 1 -- 172.21.15.6:0/1577915763 >> v1:172.21.15.6:6789/0 conn(0x7f0a3c0b1a70 legacy=0x7f0a3c142090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:13.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.361+0000 7f0a31ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:21:13.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.361+0000 7f0a31ffb640 1 monclient: found mon.c 2024-03-25T19:21:13.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.361+0000 7f0a31ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:13.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.361+0000 7f0a31ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:21:13.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.361+0000 7f0a31ffb640 1 -- 172.21.15.6:0/1577915763 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a3c141a80 con 0x7f0a3c1458a0 2024-03-25T19:21:13.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.361+0000 7f0a31ffb640 10 monclient: _finish_auth 0 2024-03-25T19:21:13.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.361+0000 7f0a31ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:21:13.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.361+0000 7f0a31ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:13.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.361+0000 7f0a436d6640 5 monclient: authenticate success, global_id 4547 2024-03-25T19:21:13.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.361+0000 7f0a436d6640 10 monclient: _renew_subs 2024-03-25T19:21:13.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.361+0000 7f0a436d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:21:13.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.361+0000 7f0a436d6640 1 -- 172.21.15.6:0/1577915763 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a3c141d30 con 0x7f0a3c1458a0 2024-03-25T19:21:13.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.361+0000 7f0a436d6640 10 client.4547.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:21:13.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.361+0000 7f0a436d6640 10 monclient: _renew_subs 2024-03-25T19:21:13.363 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.361+0000 7f0a436d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:21:13.363 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.361+0000 7f0a436d6640 1 -- 172.21.15.6:0/1577915763 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a3c1db320 con 0x7f0a3c1458a0 2024-03-25T19:21:13.363 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.361+0000 7f0a436d6640 1 librados: init done 2024-03-25T19:21:13.363 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.362+0000 7f0a436d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:13.363 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.362+0000 7f0a436d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:13.363 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.362+0000 7f0a436d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:21:13.363 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.362+0000 7f0a436d6640 1 -- 172.21.15.6:0/1577915763 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a3c0688c0 con 0x7f0a3c1458a0 2024-03-25T19:21:13.673 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.673+0000 7f0a31ffb640 1 -- 172.21.15.6:0/1577915763 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a2c0038a0 con 0x7f0a3c1458a0 2024-03-25T19:21:13.673 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.673+0000 7f0a31ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:13.673 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.673+0000 7f0a31ffb640 1 -- 172.21.15.6:0/1577915763 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f0a2c003cb0 con 0x7f0a3c1458a0 2024-03-25T19:21:13.673 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.673+0000 7f0a31ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:13.673 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.673+0000 7f0a31ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:21:13.673 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.673+0000 7f0a31ffb640 10 monclient: dump: 2024-03-25T19:21:13.673 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:21:13.674 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:13.674 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:13.674 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:13.674 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:13.674 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:13.674 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:21:13.674 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:21:13.674 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:21:13.674 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:13.674 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.673+0000 7f0a337fe640 4 set_mon_vals no callback set 2024-03-25T19:21:13.675 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.674+0000 7f0a31ffb640 1 -- 172.21.15.6:0/1577915763 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f0a2c003ed0 con 0x7f0a3c1458a0 2024-03-25T19:21:13.675 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.674+0000 7f0a41472640 10 monclient: get_auth_request con 0x7f0a2006ddf0 auth_method 0 2024-03-25T19:21:13.675 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.674+0000 7f0a31ffb640 10 client.4547.objecter ms_handle_connect 0x7f0a2006ddf0 2024-03-25T19:21:13.680 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.680+0000 7f0a31ffb640 1 -- 172.21.15.6:0/1577915763 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f0a2c054900 con 0x7f0a3c1458a0 2024-03-25T19:21:13.680 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.680+0000 7f0a31ffb640 10 client.4547.objecter ms_dispatch 0x7f0a3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:21:13.680 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.680+0000 7f0a31ffb640 3 client.4547.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:21:13.680 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.680+0000 7f0a31ffb640 3 client.4547.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:21:13.680 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.680+0000 7f0a31ffb640 20 client.4547.objecter dump_active .. 0 homeless 2024-03-25T19:21:13.680 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.680+0000 7f0a31ffb640 1 -- 172.21.15.6:0/1577915763 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0a2c050410 con 0x7f0a3c1458a0 2024-03-25T19:21:13.680 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.680+0000 7f0a31ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:13.681 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.680+0000 7f0a31ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:21:13.936 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.936+0000 7f0a436d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:13.936 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.936+0000 7f0a436d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:13.936 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.936+0000 7f0a436d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:21:13.936 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.936+0000 7f0a436d6640 1 -- 172.21.15.6:0/1577915763 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0a3c1db5d0 con 0x7f0a3c1458a0 2024-03-25T19:21:13.970 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.970+0000 7f0a31ffb640 1 -- 172.21.15.6:0/1577915763 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f0a2c052340 con 0x7f0a3c1458a0 2024-03-25T19:21:13.970 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.970+0000 7f0a31ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:13.970 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.970+0000 7f0a31ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:21:13.970 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:21:13.970 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:21:13.973 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.972+0000 7f0a436d6640 10 librados: watch_flush enter 2024-03-25T19:21:13.973 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.972+0000 7f0a436d6640 10 librados: watch_flush exit 2024-03-25T19:21:13.973 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.972+0000 7f0a436d6640 20 client.4547.objecter shutdown clearing up homeless session... 2024-03-25T19:21:13.973 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.972+0000 7f0a436d6640 10 client.4547.objecter successfully canceled tick 2024-03-25T19:21:13.973 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.973+0000 7f0a436d6640 1 -- 172.21.15.6:0/1577915763 >> v1:172.21.15.6:6800/2936652650 conn(0x7f0a2006ddf0 legacy=0x7f0a200702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:13.974 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.973+0000 7f0a436d6640 10 monclient: shutdown 2024-03-25T19:21:13.974 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.973+0000 7f0a436d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:13.974 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.973+0000 7f0a436d6640 1 -- 172.21.15.6:0/1577915763 >> v1:172.21.15.6:6790/0 conn(0x7f0a3c1458a0 legacy=0x7f0a3c1daa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:13.974 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.973+0000 7f0a436d6640 1 -- 172.21.15.6:0/1577915763 shutdown_connections 2024-03-25T19:21:13.974 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.973+0000 7f0a436d6640 1 -- 172.21.15.6:0/1577915763 >> 172.21.15.6:0/1577915763 conn(0x7f0a3c0b28f0 msgr2=0x7f0a3c06bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:13.974 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.973+0000 7f0a436d6640 1 -- 172.21.15.6:0/1577915763 shutdown_connections 2024-03-25T19:21:13.974 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.973+0000 7f0a436d6640 1 -- 172.21.15.6:0/1577915763 wait complete. 2024-03-25T19:21:13.974 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:13.973+0000 7f0a436d6640 1 librados: shutdown 2024-03-25T19:21:13.984 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:21:20.790 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:21:20.908 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.907+0000 7fb733455640 10 monclient: get_monmap_and_config 2024-03-25T19:21:20.908 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.907+0000 7fb733455640 10 monclient: build_initial_monmap 2024-03-25T19:21:20.908 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.907+0000 7fb733455640 10 monclient: monmap: 2024-03-25T19:21:20.908 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:21:20.908 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:20.908 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:21:20.909235+0000 2024-03-25T19:21:20.908 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:21:20.909235+0000 2024-03-25T19:21:20.908 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:20.909 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:20.909 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:21:20.909 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:21:20.909 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:21:20.909 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:20.910 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.907+0000 7fb733455640 1 Processor -- start 2024-03-25T19:21:20.910 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.908+0000 7fb733455640 1 -- start start 2024-03-25T19:21:20.910 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.908+0000 7fb733455640 10 monclient: init 2024-03-25T19:21:20.910 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.908+0000 7fb733455640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:20.910 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.908+0000 7fb733455640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:21:20.911 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.909+0000 7fb733455640 10 monclient(hunting): picked mon.noname-b con 0x7fb72c0b4ff0 addr v1:172.21.15.6:6790/0 2024-03-25T19:21:20.911 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.909+0000 7fb733455640 10 monclient(hunting): picked mon.noname-a con 0x7fb72c0b4670 addr v1:172.21.15.6:6789/0 2024-03-25T19:21:20.911 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.909+0000 7fb733455640 10 monclient(hunting): picked mon.noname-c con 0x7fb72c13c480 addr v1:172.21.15.192:6789/0 2024-03-25T19:21:20.911 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.909+0000 7fb733455640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb72c0ab0e0 con 0x7fb72c0b4670 2024-03-25T19:21:20.911 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.909+0000 7fb733455640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb72c0ab3c0 con 0x7fb72c13c480 2024-03-25T19:21:20.911 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.909+0000 7fb733455640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb72c0a9f30 con 0x7fb72c0b4ff0 2024-03-25T19:21:20.911 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.909+0000 7fb733455640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:20.911 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.909+0000 7fb733455640 10 monclient(hunting): authenticate will time out at 6385.149872s 2024-03-25T19:21:20.913 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.913+0000 7fb7311f1640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fb72c0b4ff0 0x7fb72c13bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42694/0 (socket says 172.21.15.6:42694) 2024-03-25T19:21:20.914 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.913+0000 7fb7311f1640 1 -- 172.21.15.6:0/963388736 learned_addr learned my addr 172.21.15.6:0/963388736 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:21:20.932 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.931+0000 7fb723fff640 1 -- 172.21.15.6:0/963388736 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1683889687 0 0) 0x7fb72c0a9f30 con 0x7fb72c0b4ff0 2024-03-25T19:21:20.933 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.931+0000 7fb723fff640 10 monclient(hunting): _negotiate 2024-03-25T19:21:20.933 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.931+0000 7fb723fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:20.933 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.931+0000 7fb723fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:20.933 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.931+0000 7fb723fff640 10 monclient(hunting): my global_id is 4550 2024-03-25T19:21:20.933 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.931+0000 7fb723fff640 1 -- 172.21.15.6:0/963388736 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb718003520 con 0x7fb72c0b4ff0 2024-03-25T19:21:20.942 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.941+0000 7fb723fff640 1 -- 172.21.15.6:0/963388736 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3387824815 0 0) 0x7fb72c0ab3c0 con 0x7fb72c13c480 2024-03-25T19:21:20.942 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.941+0000 7fb723fff640 10 monclient(hunting): _negotiate 2024-03-25T19:21:20.942 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.941+0000 7fb723fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:20.942 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.941+0000 7fb723fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:20.942 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.941+0000 7fb723fff640 10 monclient(hunting): my global_id is 4555 2024-03-25T19:21:20.942 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.941+0000 7fb723fff640 1 -- 172.21.15.6:0/963388736 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb72c0a9f30 con 0x7fb72c13c480 2024-03-25T19:21:20.951 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.950+0000 7fb723fff640 1 -- 172.21.15.6:0/963388736 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1120923296 0 0) 0x7fb72c0ab0e0 con 0x7fb72c0b4670 2024-03-25T19:21:20.952 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.950+0000 7fb723fff640 10 monclient(hunting): _negotiate 2024-03-25T19:21:20.952 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.950+0000 7fb723fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:20.952 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.950+0000 7fb723fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:20.952 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.950+0000 7fb723fff640 10 monclient(hunting): my global_id is 4548 2024-03-25T19:21:20.952 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.950+0000 7fb723fff640 1 -- 172.21.15.6:0/963388736 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb72c0ab3c0 con 0x7fb72c0b4670 2024-03-25T19:21:20.961 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.959+0000 7fb723fff640 1 -- 172.21.15.6:0/963388736 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1826329387 0 0) 0x7fb72c0a9f30 con 0x7fb72c13c480 2024-03-25T19:21:20.961 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.959+0000 7fb723fff640 1 -- 172.21.15.6:0/963388736 >> v1:172.21.15.6:6790/0 conn(0x7fb72c0b4ff0 legacy=0x7fb72c13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:20.962 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.959+0000 7fb723fff640 1 -- 172.21.15.6:0/963388736 >> v1:172.21.15.6:6789/0 conn(0x7fb72c0b4670 legacy=0x7fb72c1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:20.962 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.959+0000 7fb723fff640 10 monclient: _finish_hunting 0 2024-03-25T19:21:20.962 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.959+0000 7fb723fff640 1 monclient: found mon.noname-c 2024-03-25T19:21:20.962 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.959+0000 7fb723fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:20.962 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.959+0000 7fb723fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:21:20.962 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.959+0000 7fb723fff640 1 -- 172.21.15.6:0/963388736 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb72c0a9b70 con 0x7fb72c13c480 2024-03-25T19:21:20.962 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.959+0000 7fb723fff640 10 monclient: _finish_auth 0 2024-03-25T19:21:20.962 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.959+0000 7fb723fff640 10 monclient: _check_auth_tickets 2024-03-25T19:21:20.962 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.959+0000 7fb723fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:20.962 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.959+0000 7fb723fff640 1 -- 172.21.15.6:0/963388736 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fb7240019f0 con 0x7fb72c13c480 2024-03-25T19:21:20.962 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.959+0000 7fb723fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:20.962 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.960+0000 7fb723fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:21:20.963 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.960+0000 7fb723fff640 10 monclient: dump: 2024-03-25T19:21:20.963 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:21:20.963 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:20.963 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:20.963 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:20.963 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:20.963 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:20.963 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:21:20.963 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:21:20.963 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:21:20.963 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:20.963 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.960+0000 7fb733455640 5 monclient: authenticate success, global_id 4555 2024-03-25T19:21:20.963 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.960+0000 7fb733455640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:21:20.987 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.985+0000 7fb723fff640 1 -- 172.21.15.6:0/963388736 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb724001b50 con 0x7fb72c13c480 2024-03-25T19:21:20.987 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.985+0000 7fb723fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:20.987 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.985+0000 7fb723fff640 1 -- 172.21.15.6:0/963388736 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fb724001f60 con 0x7fb72c13c480 2024-03-25T19:21:20.987 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.985+0000 7fb723fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:20.987 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.985+0000 7fb723fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:21:20.987 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.985+0000 7fb723fff640 10 monclient: dump: 2024-03-25T19:21:20.987 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:21:20.987 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:20.987 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:20.987 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:20.987 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:20.987 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:20.988 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:21:20.988 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:21:20.988 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:21:20.988 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:20.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.985+0000 7fb733455640 10 monclient: get_monmap_and_config success 2024-03-25T19:21:20.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.985+0000 7fb733455640 4 set_mon_vals no callback set 2024-03-25T19:21:20.990 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.986+0000 7fb733455640 10 monclient: shutdown 2024-03-25T19:21:20.990 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.986+0000 7fb733455640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:20.990 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.986+0000 7fb733455640 1 -- 172.21.15.6:0/963388736 >> v1:172.21.15.192:6789/0 conn(0x7fb72c13c480 legacy=0x7fb72c14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:20.990 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.986+0000 7fb733455640 1 -- 172.21.15.6:0/963388736 shutdown_connections 2024-03-25T19:21:20.990 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.986+0000 7fb733455640 1 -- 172.21.15.6:0/963388736 >> 172.21.15.6:0/963388736 conn(0x7fb72c0617a0 msgr2=0x7fb72c061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:20.990 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.986+0000 7fb733455640 1 -- 172.21.15.6:0/963388736 shutdown_connections 2024-03-25T19:21:20.990 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.986+0000 7fb733455640 1 -- 172.21.15.6:0/963388736 wait complete. 2024-03-25T19:21:20.990 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.987+0000 7fb733455640 10 monclient: build_initial_monmap 2024-03-25T19:21:20.990 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.987+0000 7fb733455640 10 monclient: monmap: 2024-03-25T19:21:20.990 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:21:20.991 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:21:20.991 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:21:20.991 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:21:20.991 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:20.991 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:21:20.991 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:21:20.991 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:21:20.991 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:21:20.991 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:20.991 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.988+0000 7fb733455640 1 librados: starting msgr at 2024-03-25T19:21:20.991 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.988+0000 7fb733455640 1 librados: starting objecter 2024-03-25T19:21:20.991 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.988+0000 7fb733455640 1 Processor -- start 2024-03-25T19:21:20.991 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.988+0000 7fb733455640 1 -- start start 2024-03-25T19:21:20.991 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.988+0000 7fb733455640 1 librados: setting wanted keys 2024-03-25T19:21:20.991 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.988+0000 7fb733455640 1 librados: calling monclient init 2024-03-25T19:21:20.992 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.988+0000 7fb733455640 10 monclient: init 2024-03-25T19:21:20.992 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.988+0000 7fb733455640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:20.992 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.988+0000 7fb733455640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:21:20.992 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.988+0000 7fb733455640 10 monclient(hunting): picked mon.noname-c con 0x7fb72c0b4670 addr v1:172.21.15.6:6790/0 2024-03-25T19:21:20.992 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.988+0000 7fb733455640 10 monclient(hunting): picked mon.noname-a con 0x7fb72c0b4ff0 addr v1:172.21.15.6:6789/0 2024-03-25T19:21:20.992 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.988+0000 7fb733455640 10 monclient(hunting): picked mon.noname-b con 0x7fb72c13c480 addr v1:172.21.15.192:6789/0 2024-03-25T19:21:20.992 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.989+0000 7fb733455640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb72c13e6e0 con 0x7fb72c0b4ff0 2024-03-25T19:21:20.992 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.989+0000 7fb733455640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb72c13da20 con 0x7fb72c13c480 2024-03-25T19:21:20.992 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.989+0000 7fb733455640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb72c13dd00 con 0x7fb72c0b4670 2024-03-25T19:21:20.992 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.989+0000 7fb733455640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:20.992 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.989+0000 7fb733455640 10 monclient(hunting): authenticate will time out at 6385.229878s 2024-03-25T19:21:20.994 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.993+0000 7fb7309f0640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb72c0b4ff0 0x7fb72c13ebc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59950/0 (socket says 172.21.15.6:59950) 2024-03-25T19:21:20.994 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.993+0000 7fb7309f0640 1 -- 172.21.15.6:0/3716679747 learned_addr learned my addr 172.21.15.6:0/3716679747 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:21:20.998 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.997+0000 7fb721ffb640 10 client.?.objecter ms_handle_connect 0x7fb72c0b4ff0 2024-03-25T19:21:20.998 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:20.997+0000 7fb721ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:21.006 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.005+0000 7fb721ffb640 1 -- 172.21.15.6:0/3716679747 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4108460571 0 0) 0x7fb72c13e6e0 con 0x7fb72c0b4ff0 2024-03-25T19:21:21.006 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.005+0000 7fb721ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:21:21.006 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.005+0000 7fb721ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:21.007 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.005+0000 7fb721ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:21.007 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.005+0000 7fb721ffb640 10 monclient(hunting): my global_id is 4551 2024-03-25T19:21:21.007 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.005+0000 7fb721ffb640 1 -- 172.21.15.6:0/3716679747 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb710004eb0 con 0x7fb72c0b4ff0 2024-03-25T19:21:21.020 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.019+0000 7fb721ffb640 10 client.?.objecter ms_handle_connect 0x7fb72c13c480 2024-03-25T19:21:21.020 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.019+0000 7fb721ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:21.029 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.028+0000 7fb721ffb640 10 client.?.objecter ms_handle_connect 0x7fb72c0b4670 2024-03-25T19:21:21.029 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.028+0000 7fb721ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:21.033 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.032+0000 7fb721ffb640 1 -- 172.21.15.6:0/3716679747 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2117933614 0 0) 0x7fb72c13da20 con 0x7fb72c13c480 2024-03-25T19:21:21.033 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.032+0000 7fb721ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:21:21.033 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.032+0000 7fb721ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:21.033 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.032+0000 7fb721ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:21.033 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.032+0000 7fb721ffb640 10 monclient(hunting): my global_id is 4558 2024-03-25T19:21:21.033 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.032+0000 7fb721ffb640 1 -- 172.21.15.6:0/3716679747 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb72c13e6e0 con 0x7fb72c13c480 2024-03-25T19:21:21.038 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.037+0000 7fb721ffb640 1 -- 172.21.15.6:0/3716679747 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2024545433 0 0) 0x7fb710004eb0 con 0x7fb72c0b4ff0 2024-03-25T19:21:21.038 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.037+0000 7fb721ffb640 1 -- 172.21.15.6:0/3716679747 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb72c13da20 con 0x7fb72c0b4ff0 2024-03-25T19:21:21.038 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.037+0000 7fb721ffb640 1 -- 172.21.15.6:0/3716679747 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fb71c001aa0 con 0x7fb72c0b4ff0 2024-03-25T19:21:21.038 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.037+0000 7fb721ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:21.038 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.037+0000 7fb721ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:21:21.039 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.037+0000 7fb721ffb640 10 monclient(hunting): dump: 2024-03-25T19:21:21.039 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:21:21.039 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:21.039 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:21.039 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:21.039 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:21.039 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:21.039 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:21:21.039 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:21:21.039 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:21:21.039 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:21.039 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.037+0000 7fb721ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:21:21.048 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.047+0000 7fb721ffb640 1 -- 172.21.15.6:0/3716679747 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2609002830 0 0) 0x7fb72c13dd00 con 0x7fb72c0b4670 2024-03-25T19:21:21.048 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.047+0000 7fb721ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:21:21.048 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.047+0000 7fb721ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:21.048 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.047+0000 7fb721ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:21.049 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.047+0000 7fb721ffb640 10 monclient(hunting): my global_id is 4553 2024-03-25T19:21:21.049 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.047+0000 7fb721ffb640 1 -- 172.21.15.6:0/3716679747 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb710004eb0 con 0x7fb72c0b4670 2024-03-25T19:21:21.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.058+0000 7fb721ffb640 1 -- 172.21.15.6:0/3716679747 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2192534992 0 0) 0x7fb72c13e6e0 con 0x7fb72c13c480 2024-03-25T19:21:21.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.058+0000 7fb721ffb640 1 -- 172.21.15.6:0/3716679747 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb72c13dd00 con 0x7fb72c13c480 2024-03-25T19:21:21.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.058+0000 7fb721ffb640 1 -- 172.21.15.6:0/3716679747 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fb724001e60 con 0x7fb72c13c480 2024-03-25T19:21:21.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.058+0000 7fb721ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:21.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.058+0000 7fb721ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:21:21.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.058+0000 7fb721ffb640 10 monclient(hunting): dump: 2024-03-25T19:21:21.059 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:21:21.059 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:21.059 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:21.059 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:21.059 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:21.060 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:21.060 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:21:21.060 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:21:21.060 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:21:21.060 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:21.061 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.059+0000 7fb721ffb640 1 -- 172.21.15.6:0/3716679747 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1842359159 0 0) 0x7fb72c13da20 con 0x7fb72c0b4ff0 2024-03-25T19:21:21.061 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.059+0000 7fb721ffb640 1 -- 172.21.15.6:0/3716679747 >> v1:172.21.15.6:6790/0 conn(0x7fb72c0b4670 legacy=0x7fb72c130200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:21.061 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.059+0000 7fb721ffb640 1 -- 172.21.15.6:0/3716679747 >> v1:172.21.15.192:6789/0 conn(0x7fb72c13c480 legacy=0x7fb72c13f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:21.062 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.059+0000 7fb721ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:21:21.062 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.059+0000 7fb721ffb640 1 monclient: found mon.a 2024-03-25T19:21:21.062 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.059+0000 7fb721ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:21.062 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.059+0000 7fb721ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:21:21.062 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.059+0000 7fb721ffb640 1 -- 172.21.15.6:0/3716679747 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb72c13dfe0 con 0x7fb72c0b4ff0 2024-03-25T19:21:21.062 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.059+0000 7fb721ffb640 10 monclient: _finish_auth 0 2024-03-25T19:21:21.062 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.059+0000 7fb721ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:21:21.062 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.059+0000 7fb721ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:21.062 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.059+0000 7fb733455640 5 monclient: authenticate success, global_id 4551 2024-03-25T19:21:21.062 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.059+0000 7fb733455640 10 monclient: _renew_subs 2024-03-25T19:21:21.062 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.059+0000 7fb733455640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:21:21.062 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.059+0000 7fb733455640 1 -- 172.21.15.6:0/3716679747 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb72c145ad0 con 0x7fb72c0b4ff0 2024-03-25T19:21:21.062 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.059+0000 7fb733455640 10 client.4551.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:21:21.063 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.059+0000 7fb733455640 10 monclient: _renew_subs 2024-03-25T19:21:21.063 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.059+0000 7fb733455640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:21:21.063 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.059+0000 7fb733455640 1 -- 172.21.15.6:0/3716679747 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb72c13d7f0 con 0x7fb72c0b4ff0 2024-03-25T19:21:21.063 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.059+0000 7fb733455640 1 librados: init done 2024-03-25T19:21:21.063 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.060+0000 7fb733455640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:21.063 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.060+0000 7fb733455640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:21.063 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.060+0000 7fb733455640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:21:21.063 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.060+0000 7fb733455640 1 -- 172.21.15.6:0/3716679747 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb72c0b3a90 con 0x7fb72c0b4ff0 2024-03-25T19:21:21.458 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.456+0000 7fb721ffb640 1 -- 172.21.15.6:0/3716679747 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb71c001d00 con 0x7fb72c0b4ff0 2024-03-25T19:21:21.458 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.456+0000 7fb721ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:21.458 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.456+0000 7fb721ffb640 1 -- 172.21.15.6:0/3716679747 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fb71c003a60 con 0x7fb72c0b4ff0 2024-03-25T19:21:21.458 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.456+0000 7fb721ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:21.458 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.456+0000 7fb721ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:21:21.458 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.456+0000 7fb721ffb640 10 monclient: dump: 2024-03-25T19:21:21.458 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:21:21.459 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:21.459 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:21.459 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:21.459 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:21.459 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:21.459 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:21:21.459 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:21:21.459 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:21:21.459 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:21.459 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.456+0000 7fb7237fe640 4 set_mon_vals no callback set 2024-03-25T19:21:21.460 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.457+0000 7fb721ffb640 1 -- 172.21.15.6:0/3716679747 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7fb71c003c80 con 0x7fb72c0b4ff0 2024-03-25T19:21:21.460 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.458+0000 7fb7311f1640 10 monclient: get_auth_request con 0x7fb71006df90 auth_method 0 2024-03-25T19:21:21.460 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.458+0000 7fb721ffb640 10 client.4551.objecter ms_handle_connect 0x7fb71006df90 2024-03-25T19:21:21.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.465+0000 7fb721ffb640 1 -- 172.21.15.6:0/3716679747 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7fb71c054900 con 0x7fb72c0b4ff0 2024-03-25T19:21:21.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.466+0000 7fb721ffb640 10 client.4551.objecter ms_dispatch 0x7fb72c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:21:21.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.466+0000 7fb721ffb640 3 client.4551.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:21:21.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.466+0000 7fb721ffb640 3 client.4551.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:21:21.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.466+0000 7fb721ffb640 20 client.4551.objecter dump_active .. 0 homeless 2024-03-25T19:21:21.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.466+0000 7fb721ffb640 1 -- 172.21.15.6:0/3716679747 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb71c050240 con 0x7fb72c0b4ff0 2024-03-25T19:21:21.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.466+0000 7fb721ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:21.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.466+0000 7fb721ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:21:21.715 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.714+0000 7fb733455640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:21.715 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.714+0000 7fb733455640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:21.715 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.714+0000 7fb733455640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:21:21.715 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.714+0000 7fb733455640 1 -- 172.21.15.6:0/3716679747 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb72c146790 con 0x7fb72c0b4ff0 2024-03-25T19:21:21.780 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.779+0000 7fb721ffb640 1 -- 172.21.15.6:0/3716679747 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7fb71c052340 con 0x7fb72c0b4ff0 2024-03-25T19:21:21.781 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.779+0000 7fb721ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:21.781 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.779+0000 7fb721ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:21:21.781 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:21:21.781 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:21:21.784 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.782+0000 7fb733455640 10 librados: watch_flush enter 2024-03-25T19:21:21.784 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.782+0000 7fb733455640 10 librados: watch_flush exit 2024-03-25T19:21:21.785 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.782+0000 7fb733455640 20 client.4551.objecter shutdown clearing up homeless session... 2024-03-25T19:21:21.785 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.782+0000 7fb733455640 10 client.4551.objecter successfully canceled tick 2024-03-25T19:21:21.785 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.783+0000 7fb733455640 1 -- 172.21.15.6:0/3716679747 >> v1:172.21.15.6:6800/2936652650 conn(0x7fb71006df90 legacy=0x7fb710070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:21.785 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.783+0000 7fb733455640 10 monclient: shutdown 2024-03-25T19:21:21.785 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.783+0000 7fb733455640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:21.785 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.783+0000 7fb733455640 1 -- 172.21.15.6:0/3716679747 >> v1:172.21.15.6:6789/0 conn(0x7fb72c0b4ff0 legacy=0x7fb72c13ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:21.785 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.783+0000 7fb733455640 1 -- 172.21.15.6:0/3716679747 shutdown_connections 2024-03-25T19:21:21.785 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.783+0000 7fb733455640 1 -- 172.21.15.6:0/3716679747 >> 172.21.15.6:0/3716679747 conn(0x7fb72c0617a0 msgr2=0x7fb72c1405c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:21.785 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.783+0000 7fb733455640 1 -- 172.21.15.6:0/3716679747 shutdown_connections 2024-03-25T19:21:21.785 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.783+0000 7fb733455640 1 -- 172.21.15.6:0/3716679747 wait complete. 2024-03-25T19:21:21.785 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:21.783+0000 7fb733455640 1 librados: shutdown 2024-03-25T19:21:21.797 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:21:28.603 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:21:28.715 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.715+0000 7f189ccd3640 10 monclient: get_monmap_and_config 2024-03-25T19:21:28.715 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.715+0000 7f189ccd3640 10 monclient: build_initial_monmap 2024-03-25T19:21:28.715 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.715+0000 7f189ccd3640 10 monclient: monmap: 2024-03-25T19:21:28.715 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:21:28.715 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:28.716 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:21:28.716412+0000 2024-03-25T19:21:28.716 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:21:28.716412+0000 2024-03-25T19:21:28.716 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:28.716 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:28.716 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:21:28.716 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:21:28.716 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:21:28.716 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:28.717 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.716+0000 7f189ccd3640 1 Processor -- start 2024-03-25T19:21:28.717 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.716+0000 7f189ccd3640 1 -- start start 2024-03-25T19:21:28.718 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.716+0000 7f189ccd3640 10 monclient: init 2024-03-25T19:21:28.718 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.717+0000 7f189ccd3640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:28.718 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.717+0000 7f189ccd3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:21:28.718 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.717+0000 7f189ccd3640 10 monclient(hunting): picked mon.noname-a con 0x7f18980b4e70 addr v1:172.21.15.6:6789/0 2024-03-25T19:21:28.718 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.717+0000 7f189ccd3640 10 monclient(hunting): picked mon.noname-c con 0x7f18980b44b0 addr v1:172.21.15.192:6789/0 2024-03-25T19:21:28.718 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.717+0000 7f189ccd3640 10 monclient(hunting): picked mon.noname-b con 0x7f18980b1b00 addr v1:172.21.15.6:6790/0 2024-03-25T19:21:28.718 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.717+0000 7f189ccd3640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18980a96f0 con 0x7f18980b4e70 2024-03-25T19:21:28.718 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.717+0000 7f189ccd3640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18980ab0e0 con 0x7f18980b44b0 2024-03-25T19:21:28.718 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.717+0000 7f189ccd3640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1898145f70 con 0x7f18980b1b00 2024-03-25T19:21:28.718 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.717+0000 7f189ccd3640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:28.718 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.717+0000 7f189ccd3640 10 monclient(hunting): authenticate will time out at 6392.957230s 2024-03-25T19:21:28.719 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.719+0000 7f1896d9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f18980b1b00 0x7f189814dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:49220/0 (socket says 172.21.15.6:49220) 2024-03-25T19:21:28.719 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.719+0000 7f1896d9d640 1 -- 172.21.15.6:0/2935160958 learned_addr learned my addr 172.21.15.6:0/2935160958 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:21:28.730 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.730+0000 7f189559a640 1 -- 172.21.15.6:0/2935160958 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2050623364 0 0) 0x7f1898145f70 con 0x7f18980b1b00 2024-03-25T19:21:28.730 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.730+0000 7f189559a640 10 monclient(hunting): _negotiate 2024-03-25T19:21:28.730 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.730+0000 7f189559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:28.731 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.730+0000 7f189559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:28.731 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.730+0000 7f189559a640 10 monclient(hunting): my global_id is 4556 2024-03-25T19:21:28.731 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.730+0000 7f189559a640 1 -- 172.21.15.6:0/2935160958 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1880003520 con 0x7f18980b1b00 2024-03-25T19:21:28.735 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.735+0000 7f189559a640 1 -- 172.21.15.6:0/2935160958 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 179112545 0 0) 0x7f18980ab0e0 con 0x7f18980b44b0 2024-03-25T19:21:28.735 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.735+0000 7f189559a640 10 monclient(hunting): _negotiate 2024-03-25T19:21:28.735 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.735+0000 7f189559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:28.735 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.735+0000 7f189559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:28.735 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.735+0000 7f189559a640 10 monclient(hunting): my global_id is 4561 2024-03-25T19:21:28.735 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.735+0000 7f189559a640 1 -- 172.21.15.6:0/2935160958 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1898145f70 con 0x7f18980b44b0 2024-03-25T19:21:28.743 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.743+0000 7f189559a640 1 -- 172.21.15.6:0/2935160958 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 418375799 0 0) 0x7f1898145f70 con 0x7f18980b44b0 2024-03-25T19:21:28.743 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.743+0000 7f189559a640 1 -- 172.21.15.6:0/2935160958 >> v1:172.21.15.6:6790/0 conn(0x7f18980b1b00 legacy=0x7f189814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:28.744 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.743+0000 7f189559a640 1 -- 172.21.15.6:0/2935160958 >> v1:172.21.15.6:6789/0 conn(0x7f18980b4e70 legacy=0x7f189813f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:28.744 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.743+0000 7f189559a640 10 monclient: _finish_hunting 0 2024-03-25T19:21:28.744 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.743+0000 7f189559a640 1 monclient: found mon.noname-c 2024-03-25T19:21:28.744 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.743+0000 7f189559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:28.744 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.743+0000 7f189559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:21:28.744 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.743+0000 7f189559a640 1 -- 172.21.15.6:0/2935160958 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18980a9b80 con 0x7f18980b44b0 2024-03-25T19:21:28.744 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.743+0000 7f189559a640 10 monclient: _finish_auth 0 2024-03-25T19:21:28.744 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.743+0000 7f189559a640 10 monclient: _check_auth_tickets 2024-03-25T19:21:28.744 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.743+0000 7f189559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:28.744 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.743+0000 7f189559a640 1 -- 172.21.15.6:0/2935160958 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f188c001670 con 0x7f18980b44b0 2024-03-25T19:21:28.744 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.743+0000 7f189559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:28.744 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.743+0000 7f189559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:21:28.744 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.743+0000 7f189559a640 10 monclient: dump: 2024-03-25T19:21:28.744 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:21:28.744 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:28.745 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:28.745 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:28.745 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:28.745 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:28.745 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:21:28.745 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:21:28.745 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:21:28.745 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:28.745 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.743+0000 7f189ccd3640 5 monclient: authenticate success, global_id 4561 2024-03-25T19:21:28.745 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.743+0000 7f189ccd3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:21:28.754 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.754+0000 7f189559a640 1 -- 172.21.15.6:0/2935160958 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f188c0017d0 con 0x7f18980b44b0 2024-03-25T19:21:28.755 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.754+0000 7f189559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:28.755 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.754+0000 7f189559a640 1 -- 172.21.15.6:0/2935160958 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f188c001b60 con 0x7f18980b44b0 2024-03-25T19:21:28.755 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.754+0000 7f189559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:28.755 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.754+0000 7f189559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:21:28.755 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.754+0000 7f189559a640 10 monclient: dump: 2024-03-25T19:21:28.755 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:21:28.755 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:28.755 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:28.755 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:28.755 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:28.755 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:28.755 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:21:28.756 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:21:28.756 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:21:28.756 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:28.756 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.755+0000 7f189ccd3640 10 monclient: get_monmap_and_config success 2024-03-25T19:21:28.756 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.755+0000 7f189ccd3640 4 set_mon_vals no callback set 2024-03-25T19:21:28.758 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.755+0000 7f189ccd3640 10 monclient: shutdown 2024-03-25T19:21:28.758 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.755+0000 7f189ccd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:28.758 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.755+0000 7f189ccd3640 1 -- 172.21.15.6:0/2935160958 >> v1:172.21.15.192:6789/0 conn(0x7f18980b44b0 legacy=0x7f1898145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:28.758 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.755+0000 7f189ccd3640 1 -- 172.21.15.6:0/2935160958 shutdown_connections 2024-03-25T19:21:28.758 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.755+0000 7f189ccd3640 1 -- 172.21.15.6:0/2935160958 >> 172.21.15.6:0/2935160958 conn(0x7f18980617a0 msgr2=0x7f1898061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:28.758 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.755+0000 7f189ccd3640 1 -- 172.21.15.6:0/2935160958 shutdown_connections 2024-03-25T19:21:28.758 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.755+0000 7f189ccd3640 1 -- 172.21.15.6:0/2935160958 wait complete. 2024-03-25T19:21:28.759 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.756+0000 7f189ccd3640 10 monclient: build_initial_monmap 2024-03-25T19:21:28.759 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.756+0000 7f189ccd3640 10 monclient: monmap: 2024-03-25T19:21:28.759 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:21:28.759 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:21:28.759 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:21:28.759 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:21:28.759 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:28.759 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:21:28.759 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:21:28.759 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:21:28.759 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:21:28.759 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:28.759 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.756+0000 7f189ccd3640 1 librados: starting msgr at 2024-03-25T19:21:28.760 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.756+0000 7f189ccd3640 1 librados: starting objecter 2024-03-25T19:21:28.760 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.757+0000 7f189ccd3640 1 Processor -- start 2024-03-25T19:21:28.760 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.757+0000 7f189ccd3640 1 -- start start 2024-03-25T19:21:28.760 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.757+0000 7f189ccd3640 1 librados: setting wanted keys 2024-03-25T19:21:28.760 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.757+0000 7f189ccd3640 1 librados: calling monclient init 2024-03-25T19:21:28.760 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.757+0000 7f189ccd3640 10 monclient: init 2024-03-25T19:21:28.760 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.757+0000 7f189ccd3640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:28.760 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.757+0000 7f189ccd3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:21:28.760 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.757+0000 7f189ccd3640 10 monclient(hunting): picked mon.noname-c con 0x7f18980b1b00 addr v1:172.21.15.6:6790/0 2024-03-25T19:21:28.760 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.757+0000 7f189ccd3640 10 monclient(hunting): picked mon.noname-b con 0x7f18980b44b0 addr v1:172.21.15.192:6789/0 2024-03-25T19:21:28.760 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.757+0000 7f189ccd3640 10 monclient(hunting): picked mon.noname-a con 0x7f18980b4e70 addr v1:172.21.15.6:6789/0 2024-03-25T19:21:28.760 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.757+0000 7f189ccd3640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1898136b50 con 0x7f18980b4e70 2024-03-25T19:21:28.760 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.757+0000 7f189ccd3640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1898137eb0 con 0x7f18980b44b0 2024-03-25T19:21:28.760 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.757+0000 7f189ccd3640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1898134590 con 0x7f18980b1b00 2024-03-25T19:21:28.761 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.757+0000 7f189ccd3640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:28.761 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.757+0000 7f189ccd3640 10 monclient(hunting): authenticate will time out at 6392.997473s 2024-03-25T19:21:28.766 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.767+0000 7f189659c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f18980b1b00 0x7f1898135a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:49222/0 (socket says 172.21.15.6:49222) 2024-03-25T19:21:28.767 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.767+0000 7f189659c640 1 -- 172.21.15.6:0/2211462683 learned_addr learned my addr 172.21.15.6:0/2211462683 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:21:28.767 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.767+0000 7f1886ffd640 10 client.?.objecter ms_handle_connect 0x7f18980b1b00 2024-03-25T19:21:28.767 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.767+0000 7f1886ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:28.773 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.773+0000 7f1886ffd640 10 client.?.objecter ms_handle_connect 0x7f18980b44b0 2024-03-25T19:21:28.773 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.773+0000 7f1886ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:28.780 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.779+0000 7f1886ffd640 1 -- 172.21.15.6:0/2211462683 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3586365573 0 0) 0x7f1898134590 con 0x7f18980b1b00 2024-03-25T19:21:28.780 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.779+0000 7f1886ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:21:28.780 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.779+0000 7f1886ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:28.780 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.779+0000 7f1886ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:28.780 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.779+0000 7f1886ffd640 10 monclient(hunting): my global_id is 4559 2024-03-25T19:21:28.780 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.779+0000 7f1886ffd640 1 -- 172.21.15.6:0/2211462683 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1878004eb0 con 0x7f18980b1b00 2024-03-25T19:21:28.780 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.780+0000 7f1886ffd640 10 client.?.objecter ms_handle_connect 0x7f18980b4e70 2024-03-25T19:21:28.780 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.780+0000 7f1886ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:28.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.784+0000 7f1886ffd640 1 -- 172.21.15.6:0/2211462683 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 915218425 0 0) 0x7f1898137eb0 con 0x7f18980b44b0 2024-03-25T19:21:28.784 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.784+0000 7f1886ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:21:28.784 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.784+0000 7f1886ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:28.784 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.784+0000 7f1886ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:28.784 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.784+0000 7f1886ffd640 10 monclient(hunting): my global_id is 4564 2024-03-25T19:21:28.784 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.784+0000 7f1886ffd640 1 -- 172.21.15.6:0/2211462683 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1898134590 con 0x7f18980b44b0 2024-03-25T19:21:28.798 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.798+0000 7f1886ffd640 1 -- 172.21.15.6:0/2211462683 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2962706249 0 0) 0x7f1898136b50 con 0x7f18980b4e70 2024-03-25T19:21:28.798 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.798+0000 7f1886ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:21:28.798 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.798+0000 7f1886ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:28.799 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.798+0000 7f1886ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:28.799 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.798+0000 7f1886ffd640 10 monclient(hunting): my global_id is 4554 2024-03-25T19:21:28.799 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.798+0000 7f1886ffd640 1 -- 172.21.15.6:0/2211462683 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1898137eb0 con 0x7f18980b4e70 2024-03-25T19:21:28.857 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:07:46.203 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:28.865 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:07:46.212 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:28.893 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.892+0000 7f1886ffd640 1 -- 172.21.15.6:0/2211462683 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1475998530 0 0) 0x7f1898137eb0 con 0x7f18980b4e70 2024-03-25T19:21:28.893 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.892+0000 7f1886ffd640 1 -- 172.21.15.6:0/2211462683 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1898136b50 con 0x7f18980b4e70 2024-03-25T19:21:28.893 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.892+0000 7f1886ffd640 1 -- 172.21.15.6:0/2211462683 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f1890001950 con 0x7f18980b4e70 2024-03-25T19:21:28.893 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.892+0000 7f1886ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:28.893 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.892+0000 7f1886ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:21:28.893 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.892+0000 7f1886ffd640 10 monclient(hunting): dump: 2024-03-25T19:21:28.893 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:21:28.893 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:28.893 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:28.893 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:28.893 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:28.894 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:28.894 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:21:28.894 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:21:28.894 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:21:28.894 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:28.894 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.892+0000 7f1886ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:21:28.906 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:07:46.253 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:28.909 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:07:46.227 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:28.922 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.922+0000 7f1886ffd640 1 -- 172.21.15.6:0/2211462683 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3880176723 0 0) 0x7f1898134590 con 0x7f18980b44b0 2024-03-25T19:21:28.922 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.922+0000 7f1886ffd640 1 -- 172.21.15.6:0/2211462683 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1898137eb0 con 0x7f18980b44b0 2024-03-25T19:21:28.922 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.922+0000 7f1886ffd640 1 -- 172.21.15.6:0/2211462683 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f188c001ae0 con 0x7f18980b44b0 2024-03-25T19:21:28.922 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.922+0000 7f1886ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:28.922 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.922+0000 7f1886ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:21:28.922 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.922+0000 7f1886ffd640 10 monclient(hunting): dump: 2024-03-25T19:21:28.922 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:21:28.922 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:28.922 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:28.922 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:28.923 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:28.923 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:28.923 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:21:28.923 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:21:28.923 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:21:28.923 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:28.930 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.929+0000 7f1886ffd640 1 -- 172.21.15.6:0/2211462683 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3237557169 0 0) 0x7f1898136b50 con 0x7f18980b4e70 2024-03-25T19:21:28.930 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.929+0000 7f1886ffd640 1 -- 172.21.15.6:0/2211462683 >> v1:172.21.15.6:6790/0 conn(0x7f18980b1b00 legacy=0x7f1898135a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:28.930 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.929+0000 7f1886ffd640 1 -- 172.21.15.6:0/2211462683 >> v1:172.21.15.192:6789/0 conn(0x7f18980b44b0 legacy=0x7f1898136260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:28.930 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.929+0000 7f1886ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:21:28.930 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.929+0000 7f1886ffd640 1 monclient: found mon.a 2024-03-25T19:21:28.930 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.929+0000 7f1886ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:28.930 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.929+0000 7f1886ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:21:28.931 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.929+0000 7f1886ffd640 1 -- 172.21.15.6:0/2211462683 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1898134720 con 0x7f18980b4e70 2024-03-25T19:21:28.931 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.929+0000 7f1886ffd640 10 monclient: _finish_auth 0 2024-03-25T19:21:28.931 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.929+0000 7f1886ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:21:28.931 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.929+0000 7f1886ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:28.931 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.929+0000 7f189ccd3640 5 monclient: authenticate success, global_id 4554 2024-03-25T19:21:28.931 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.929+0000 7f189ccd3640 10 monclient: _renew_subs 2024-03-25T19:21:28.931 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.929+0000 7f189ccd3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:21:28.931 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.929+0000 7f189ccd3640 1 -- 172.21.15.6:0/2211462683 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1898134970 con 0x7f18980b4e70 2024-03-25T19:21:28.931 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.929+0000 7f189ccd3640 10 client.4554.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:21:28.931 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.929+0000 7f189ccd3640 10 monclient: _renew_subs 2024-03-25T19:21:28.931 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.929+0000 7f189ccd3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:21:28.931 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.929+0000 7f189ccd3640 1 -- 172.21.15.6:0/2211462683 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f18981daf30 con 0x7f18980b4e70 2024-03-25T19:21:28.931 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.929+0000 7f189ccd3640 1 librados: init done 2024-03-25T19:21:28.932 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.930+0000 7f189ccd3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:28.932 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.930+0000 7f189ccd3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:28.932 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.930+0000 7f189ccd3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:21:28.932 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:28.930+0000 7f189ccd3640 1 -- 172.21.15.6:0/2211462683 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f189804f730 con 0x7f18980b4e70 2024-03-25T19:21:28.932 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:07:46.247 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:28.935 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:07:46.282 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:28.940 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:07:46.287 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:28.963 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:07:46.281 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:28.976 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:07:46.323 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:29.239 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:29.239+0000 7f1886ffd640 1 -- 172.21.15.6:0/2211462683 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1890001d60 con 0x7f18980b4e70 2024-03-25T19:21:29.239 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:29.239+0000 7f1886ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:29.240 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:29.239+0000 7f1886ffd640 1 -- 172.21.15.6:0/2211462683 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f1890004980 con 0x7f18980b4e70 2024-03-25T19:21:29.240 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:29.239+0000 7f1886ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:29.240 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:29.239+0000 7f1886ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:21:29.240 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:29.239+0000 7f1886ffd640 10 monclient: dump: 2024-03-25T19:21:29.240 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:21:29.240 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:29.240 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:29.240 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:29.240 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:29.240 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:29.240 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:21:29.240 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:21:29.241 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:21:29.241 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:29.241 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:29.239+0000 7f1894d99640 4 set_mon_vals no callback set 2024-03-25T19:21:29.241 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:29.240+0000 7f1886ffd640 1 -- 172.21.15.6:0/2211462683 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f189001a950 con 0x7f18980b4e70 2024-03-25T19:21:29.241 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:29.241+0000 7f189659c640 10 monclient: get_auth_request con 0x7f187806de90 auth_method 0 2024-03-25T19:21:29.241 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:29.241+0000 7f1886ffd640 10 client.4554.objecter ms_handle_connect 0x7f187806de90 2024-03-25T19:21:29.250 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:29.250+0000 7f1886ffd640 1 -- 172.21.15.6:0/2211462683 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f18900545a0 con 0x7f18980b4e70 2024-03-25T19:21:29.250 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:29.250+0000 7f1886ffd640 10 client.4554.objecter ms_dispatch 0x7f1898000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:21:29.250 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:29.250+0000 7f1886ffd640 3 client.4554.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:21:29.250 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:29.250+0000 7f1886ffd640 3 client.4554.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:21:29.250 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:29.250+0000 7f1886ffd640 20 client.4554.objecter dump_active .. 0 homeless 2024-03-25T19:21:29.250 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:29.250+0000 7f1886ffd640 1 -- 172.21.15.6:0/2211462683 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f189004fee0 con 0x7f18980b4e70 2024-03-25T19:21:29.250 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:29.250+0000 7f1886ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:29.251 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:29.250+0000 7f1886ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:21:29.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:29.475+0000 7f189ccd3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:29.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:29.475+0000 7f189ccd3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:29.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:29.475+0000 7f189ccd3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:21:29.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:29.475+0000 7f189ccd3640 1 -- 172.21.15.6:0/2211462683 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1898136d00 con 0x7f18980b4e70 2024-03-25T19:21:29.532 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:29.532+0000 7f1886ffd640 1 -- 172.21.15.6:0/2211462683 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f1898136d00 con 0x7f18980b4e70 2024-03-25T19:21:29.533 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:29.533+0000 7f1886ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:29.533 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:29.533+0000 7f1886ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:21:29.533 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:21:29.533 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:21:29.536 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:29.535+0000 7f189ccd3640 10 librados: watch_flush enter 2024-03-25T19:21:29.536 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:29.535+0000 7f189ccd3640 10 librados: watch_flush exit 2024-03-25T19:21:29.536 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:29.535+0000 7f189ccd3640 20 client.4554.objecter shutdown clearing up homeless session... 2024-03-25T19:21:29.536 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:29.535+0000 7f189ccd3640 10 client.4554.objecter successfully canceled tick 2024-03-25T19:21:29.536 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:29.535+0000 7f189ccd3640 1 -- 172.21.15.6:0/2211462683 >> v1:172.21.15.6:6800/2936652650 conn(0x7f187806de90 legacy=0x7f1878070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:29.536 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:29.535+0000 7f189ccd3640 10 monclient: shutdown 2024-03-25T19:21:29.536 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:29.535+0000 7f189ccd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:29.536 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:29.535+0000 7f189ccd3640 1 -- 172.21.15.6:0/2211462683 >> v1:172.21.15.6:6789/0 conn(0x7f18980b4e70 legacy=0x7f1898133e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:29.536 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:29.535+0000 7f189ccd3640 1 -- 172.21.15.6:0/2211462683 shutdown_connections 2024-03-25T19:21:29.536 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:29.535+0000 7f189ccd3640 1 -- 172.21.15.6:0/2211462683 >> 172.21.15.6:0/2211462683 conn(0x7f18980617a0 msgr2=0x7f1898144a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:29.536 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:29.535+0000 7f189ccd3640 1 -- 172.21.15.6:0/2211462683 shutdown_connections 2024-03-25T19:21:29.536 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:29.535+0000 7f189ccd3640 1 -- 172.21.15.6:0/2211462683 wait complete. 2024-03-25T19:21:29.536 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:29.535+0000 7f189ccd3640 1 librados: shutdown 2024-03-25T19:21:29.546 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:21:36.353 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:21:36.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.471+0000 7f38f90c8640 10 monclient: get_monmap_and_config 2024-03-25T19:21:36.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.471+0000 7f38f90c8640 10 monclient: build_initial_monmap 2024-03-25T19:21:36.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.471+0000 7f38f90c8640 10 monclient: monmap: 2024-03-25T19:21:36.471 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:21:36.471 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:36.471 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:21:36.472268+0000 2024-03-25T19:21:36.472 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:21:36.472268+0000 2024-03-25T19:21:36.472 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:36.472 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:36.472 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:21:36.472 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:21:36.472 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:21:36.472 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:36.473 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.472+0000 7f38f90c8640 1 Processor -- start 2024-03-25T19:21:36.473 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.472+0000 7f38f90c8640 1 -- start start 2024-03-25T19:21:36.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.472+0000 7f38f90c8640 10 monclient: init 2024-03-25T19:21:36.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.472+0000 7f38f90c8640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:36.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.472+0000 7f38f90c8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:21:36.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.472+0000 7f38f90c8640 10 monclient(hunting): picked mon.noname-c con 0x7f38f40b4ff0 addr v1:172.21.15.192:6789/0 2024-03-25T19:21:36.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.472+0000 7f38f90c8640 10 monclient(hunting): picked mon.noname-b con 0x7f38f40b4670 addr v1:172.21.15.6:6790/0 2024-03-25T19:21:36.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.472+0000 7f38f90c8640 10 monclient(hunting): picked mon.noname-a con 0x7f38f413c480 addr v1:172.21.15.6:6789/0 2024-03-25T19:21:36.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.472+0000 7f38f90c8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f40ab0e0 con 0x7f38f413c480 2024-03-25T19:21:36.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.472+0000 7f38f90c8640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f40ab3c0 con 0x7f38f40b4ff0 2024-03-25T19:21:36.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.473+0000 7f38f90c8640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f40a9f30 con 0x7f38f40b4670 2024-03-25T19:21:36.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.473+0000 7f38f90c8640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:36.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.473+0000 7f38f90c8640 10 monclient(hunting): authenticate will time out at 6400.712790s 2024-03-25T19:21:36.480 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.480+0000 7f38f2d9d640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f38f40b4ff0 0x7f38f413bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:50774/0 (socket says 172.21.15.6:50774) 2024-03-25T19:21:36.480 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.480+0000 7f38f2d9d640 1 -- 172.21.15.6:0/425487669 learned_addr learned my addr 172.21.15.6:0/425487669 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:21:36.485 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.485+0000 7f38f259c640 1 -- 172.21.15.6:0/425487669 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2322970008 0 0) 0x7f38f40ab3c0 con 0x7f38f40b4ff0 2024-03-25T19:21:36.485 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.485+0000 7f38f259c640 10 monclient(hunting): _negotiate 2024-03-25T19:21:36.485 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.485+0000 7f38f259c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:36.485 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.485+0000 7f38f259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:36.485 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.485+0000 7f38f259c640 10 monclient(hunting): my global_id is 4567 2024-03-25T19:21:36.485 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.485+0000 7f38f259c640 1 -- 172.21.15.6:0/425487669 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38e0003520 con 0x7f38f40b4ff0 2024-03-25T19:21:36.490 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.490+0000 7f38f259c640 1 -- 172.21.15.6:0/425487669 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1526987307 0 0) 0x7f38e0003520 con 0x7f38f40b4ff0 2024-03-25T19:21:36.491 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.490+0000 7f38f259c640 1 -- 172.21.15.6:0/425487669 >> v1:172.21.15.6:6790/0 conn(0x7f38f40b4670 legacy=0x7f38f41425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:36.491 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.490+0000 7f38f259c640 1 -- 172.21.15.6:0/425487669 >> v1:172.21.15.6:6789/0 conn(0x7f38f413c480 legacy=0x7f38f414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:21:36.491 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.490+0000 7f38f259c640 10 monclient: _finish_hunting 0 2024-03-25T19:21:36.491 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.490+0000 7f38f259c640 1 monclient: found mon.noname-c 2024-03-25T19:21:36.491 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.490+0000 7f38f259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:36.491 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.490+0000 7f38f259c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:21:36.491 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.490+0000 7f38f259c640 1 -- 172.21.15.6:0/425487669 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f40a9b70 con 0x7f38f40b4ff0 2024-03-25T19:21:36.491 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.490+0000 7f38f259c640 10 monclient: _finish_auth 0 2024-03-25T19:21:36.491 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.490+0000 7f38f259c640 10 monclient: _check_auth_tickets 2024-03-25T19:21:36.491 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.490+0000 7f38f259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:36.491 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.490+0000 7f38f259c640 1 -- 172.21.15.6:0/425487669 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f38e4001690 con 0x7f38f40b4ff0 2024-03-25T19:21:36.491 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.490+0000 7f38f259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:36.491 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.490+0000 7f38f259c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:21:36.491 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.490+0000 7f38f259c640 10 monclient: dump: 2024-03-25T19:21:36.492 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:21:36.492 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:36.492 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:36.492 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:36.492 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:36.492 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:36.492 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:21:36.492 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:21:36.492 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:21:36.492 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:36.492 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.490+0000 7f38f90c8640 5 monclient: authenticate success, global_id 4567 2024-03-25T19:21:36.492 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.490+0000 7f38f90c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:21:36.500 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.500+0000 7f38f259c640 1 -- 172.21.15.6:0/425487669 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38e4001860 con 0x7f38f40b4ff0 2024-03-25T19:21:36.500 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.500+0000 7f38f259c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:36.500 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.500+0000 7f38f259c640 1 -- 172.21.15.6:0/425487669 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f38e4001cb0 con 0x7f38f40b4ff0 2024-03-25T19:21:36.500 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.500+0000 7f38f259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:36.500 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.500+0000 7f38f259c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:21:36.500 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.500+0000 7f38f259c640 10 monclient: dump: 2024-03-25T19:21:36.500 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:21:36.501 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:36.501 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:36.501 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:36.501 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:36.501 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:36.501 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:21:36.501 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:21:36.501 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:21:36.501 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:36.501 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.500+0000 7f38f90c8640 10 monclient: get_monmap_and_config success 2024-03-25T19:21:36.501 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.500+0000 7f38f90c8640 4 set_mon_vals no callback set 2024-03-25T19:21:36.504 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.501+0000 7f38f90c8640 10 monclient: shutdown 2024-03-25T19:21:36.504 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.501+0000 7f38f90c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:36.504 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.501+0000 7f38f90c8640 1 -- 172.21.15.6:0/425487669 >> v1:172.21.15.192:6789/0 conn(0x7f38f40b4ff0 legacy=0x7f38f413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:36.504 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.501+0000 7f38f90c8640 1 -- 172.21.15.6:0/425487669 shutdown_connections 2024-03-25T19:21:36.504 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.501+0000 7f38f90c8640 1 -- 172.21.15.6:0/425487669 >> 172.21.15.6:0/425487669 conn(0x7f38f40617a0 msgr2=0x7f38f4061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:36.504 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.501+0000 7f38f90c8640 1 -- 172.21.15.6:0/425487669 shutdown_connections 2024-03-25T19:21:36.504 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.501+0000 7f38f90c8640 1 -- 172.21.15.6:0/425487669 wait complete. 2024-03-25T19:21:36.505 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.502+0000 7f38f90c8640 10 monclient: build_initial_monmap 2024-03-25T19:21:36.505 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.502+0000 7f38f90c8640 10 monclient: monmap: 2024-03-25T19:21:36.505 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:21:36.505 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:21:36.505 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:21:36.505 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:21:36.505 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:36.505 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:21:36.505 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:21:36.505 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:21:36.505 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:21:36.505 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:36.505 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.502+0000 7f38f90c8640 1 librados: starting msgr at 2024-03-25T19:21:36.505 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.502+0000 7f38f90c8640 1 librados: starting objecter 2024-03-25T19:21:36.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.502+0000 7f38f90c8640 1 Processor -- start 2024-03-25T19:21:36.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.502+0000 7f38f90c8640 1 -- start start 2024-03-25T19:21:36.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.502+0000 7f38f90c8640 1 librados: setting wanted keys 2024-03-25T19:21:36.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.502+0000 7f38f90c8640 1 librados: calling monclient init 2024-03-25T19:21:36.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.502+0000 7f38f90c8640 10 monclient: init 2024-03-25T19:21:36.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.503+0000 7f38f90c8640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:36.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.503+0000 7f38f90c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:21:36.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.503+0000 7f38f90c8640 10 monclient(hunting): picked mon.noname-c con 0x7f38f40b4670 addr v1:172.21.15.6:6790/0 2024-03-25T19:21:36.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.503+0000 7f38f90c8640 10 monclient(hunting): picked mon.noname-a con 0x7f38f40b4ff0 addr v1:172.21.15.6:6789/0 2024-03-25T19:21:36.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.503+0000 7f38f90c8640 10 monclient(hunting): picked mon.noname-b con 0x7f38f413c480 addr v1:172.21.15.192:6789/0 2024-03-25T19:21:36.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.503+0000 7f38f90c8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f4144140 con 0x7f38f40b4ff0 2024-03-25T19:21:36.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.503+0000 7f38f90c8640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f4145480 con 0x7f38f413c480 2024-03-25T19:21:36.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.503+0000 7f38f90c8640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f4144390 con 0x7f38f40b4670 2024-03-25T19:21:36.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.503+0000 7f38f90c8640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:36.507 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.503+0000 7f38f90c8640 10 monclient(hunting): authenticate will time out at 6400.743440s 2024-03-25T19:21:36.511 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.511+0000 7f38ebfff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f38f40b4ff0 0x7f38f412fcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41074/0 (socket says 172.21.15.6:41074) 2024-03-25T19:21:36.511 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.511+0000 7f38ebfff640 1 -- 172.21.15.6:0/3058684053 learned_addr learned my addr 172.21.15.6:0/3058684053 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:21:36.512 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.512+0000 7f38eb7fe640 10 client.?.objecter ms_handle_connect 0x7f38f40b4670 2024-03-25T19:21:36.512 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.512+0000 7f38eb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:36.515 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.515+0000 7f38eb7fe640 10 client.?.objecter ms_handle_connect 0x7f38f40b4ff0 2024-03-25T19:21:36.515 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.515+0000 7f38eb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:36.517 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.518+0000 7f38eb7fe640 10 client.?.objecter ms_handle_connect 0x7f38f413c480 2024-03-25T19:21:36.517 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.518+0000 7f38eb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:36.524 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.523+0000 7f38eb7fe640 1 -- 172.21.15.6:0/3058684053 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1673914677 0 0) 0x7f38f4145480 con 0x7f38f413c480 2024-03-25T19:21:36.524 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.523+0000 7f38eb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:21:36.524 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.523+0000 7f38eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:36.524 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.523+0000 7f38eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:36.524 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.523+0000 7f38eb7fe640 10 monclient(hunting): my global_id is 4570 2024-03-25T19:21:36.524 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.523+0000 7f38eb7fe640 1 -- 172.21.15.6:0/3058684053 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38d4004eb0 con 0x7f38f413c480 2024-03-25T19:21:36.526 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.526+0000 7f38eb7fe640 1 -- 172.21.15.6:0/3058684053 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2176430221 0 0) 0x7f38f4144390 con 0x7f38f40b4670 2024-03-25T19:21:36.526 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.526+0000 7f38eb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:21:36.526 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.526+0000 7f38eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:36.526 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.526+0000 7f38eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:36.527 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.526+0000 7f38eb7fe640 10 monclient(hunting): my global_id is 4565 2024-03-25T19:21:36.527 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.526+0000 7f38eb7fe640 1 -- 172.21.15.6:0/3058684053 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38f4145480 con 0x7f38f40b4670 2024-03-25T19:21:36.534 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.534+0000 7f38eb7fe640 1 -- 172.21.15.6:0/3058684053 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 330557746 0 0) 0x7f38f4144140 con 0x7f38f40b4ff0 2024-03-25T19:21:36.534 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.534+0000 7f38eb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:21:36.534 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.534+0000 7f38eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:36.534 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.534+0000 7f38eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:36.534 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.534+0000 7f38eb7fe640 10 monclient(hunting): my global_id is 4557 2024-03-25T19:21:36.534 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.534+0000 7f38eb7fe640 1 -- 172.21.15.6:0/3058684053 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38f4144390 con 0x7f38f40b4ff0 2024-03-25T19:21:36.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.545+0000 7f38eb7fe640 1 -- 172.21.15.6:0/3058684053 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3923273095 0 0) 0x7f38d4004eb0 con 0x7f38f413c480 2024-03-25T19:21:36.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.545+0000 7f38eb7fe640 1 -- 172.21.15.6:0/3058684053 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38f4144140 con 0x7f38f413c480 2024-03-25T19:21:36.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.545+0000 7f38eb7fe640 1 -- 172.21.15.6:0/3058684053 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f38ec001ad0 con 0x7f38f413c480 2024-03-25T19:21:36.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.545+0000 7f38eb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:36.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.545+0000 7f38eb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:21:36.546 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.545+0000 7f38eb7fe640 10 monclient(hunting): dump: 2024-03-25T19:21:36.546 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:21:36.546 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:36.546 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:36.546 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:36.546 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:36.546 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:36.546 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:21:36.546 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:21:36.546 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:21:36.546 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:36.546 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.545+0000 7f38eb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:21:36.547 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.547+0000 7f38eb7fe640 1 -- 172.21.15.6:0/3058684053 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2322611248 0 0) 0x7f38f4145480 con 0x7f38f40b4670 2024-03-25T19:21:36.547 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.547+0000 7f38eb7fe640 1 -- 172.21.15.6:0/3058684053 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38d4004eb0 con 0x7f38f40b4670 2024-03-25T19:21:36.548 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.547+0000 7f38eb7fe640 1 -- 172.21.15.6:0/3058684053 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f38e4001b70 con 0x7f38f40b4670 2024-03-25T19:21:36.548 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.547+0000 7f38eb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:36.548 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.547+0000 7f38eb7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:21:36.548 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.547+0000 7f38eb7fe640 10 monclient(hunting): dump: 2024-03-25T19:21:36.548 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:21:36.548 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:36.548 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:36.548 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:36.548 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:36.548 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:36.548 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:21:36.548 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:21:36.548 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:21:36.548 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:36.558 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.558+0000 7f38eb7fe640 1 -- 172.21.15.6:0/3058684053 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3203587281 0 0) 0x7f38f4144390 con 0x7f38f40b4ff0 2024-03-25T19:21:36.558 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.558+0000 7f38eb7fe640 1 -- 172.21.15.6:0/3058684053 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38f4145480 con 0x7f38f40b4ff0 2024-03-25T19:21:36.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.558+0000 7f38eb7fe640 1 -- 172.21.15.6:0/3058684053 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f38dc001970 con 0x7f38f40b4ff0 2024-03-25T19:21:36.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.558+0000 7f38eb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:36.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.558+0000 7f38eb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:21:36.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.558+0000 7f38eb7fe640 10 monclient(hunting): dump: 2024-03-25T19:21:36.559 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:21:36.559 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:36.559 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:36.559 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:36.559 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:36.559 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:36.559 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:21:36.559 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:21:36.559 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:21:36.559 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:36.566 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.566+0000 7f38eb7fe640 1 -- 172.21.15.6:0/3058684053 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 69394574 0 0) 0x7f38f4144140 con 0x7f38f413c480 2024-03-25T19:21:36.567 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.566+0000 7f38eb7fe640 1 -- 172.21.15.6:0/3058684053 >> v1:172.21.15.6:6790/0 conn(0x7f38f40b4670 legacy=0x7f38f4144d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:36.567 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.566+0000 7f38eb7fe640 1 -- 172.21.15.6:0/3058684053 >> v1:172.21.15.6:6789/0 conn(0x7f38f40b4ff0 legacy=0x7f38f412fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:36.567 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.566+0000 7f38eb7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:21:36.567 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.566+0000 7f38eb7fe640 1 monclient: found mon.b 2024-03-25T19:21:36.567 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.566+0000 7f38eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:36.567 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.566+0000 7f38eb7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:21:36.567 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.566+0000 7f38eb7fe640 1 -- 172.21.15.6:0/3058684053 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f4144670 con 0x7f38f413c480 2024-03-25T19:21:36.567 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.566+0000 7f38eb7fe640 10 monclient: _finish_auth 0 2024-03-25T19:21:36.567 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.566+0000 7f38eb7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:21:36.567 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.566+0000 7f38eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:36.567 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.566+0000 7f38f90c8640 5 monclient: authenticate success, global_id 4570 2024-03-25T19:21:36.567 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.566+0000 7f38f90c8640 10 monclient: _renew_subs 2024-03-25T19:21:36.567 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.566+0000 7f38f90c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:21:36.567 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.566+0000 7f38f90c8640 1 -- 172.21.15.6:0/3058684053 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38f413eb60 con 0x7f38f413c480 2024-03-25T19:21:36.568 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.566+0000 7f38f90c8640 10 client.4570.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:21:36.568 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.566+0000 7f38f90c8640 10 monclient: _renew_subs 2024-03-25T19:21:36.568 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.566+0000 7f38f90c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:21:36.568 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.566+0000 7f38f90c8640 1 -- 172.21.15.6:0/3058684053 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f38f413e7a0 con 0x7f38f413c480 2024-03-25T19:21:36.568 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.566+0000 7f38f90c8640 1 librados: init done 2024-03-25T19:21:36.568 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.567+0000 7f38f90c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:36.568 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.567+0000 7f38f90c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:36.569 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.567+0000 7f38f90c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:21:36.569 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.567+0000 7f38f90c8640 1 -- 172.21.15.6:0/3058684053 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38f406dbd0 con 0x7f38f413c480 2024-03-25T19:21:36.574 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.574+0000 7f38eb7fe640 1 -- 172.21.15.6:0/3058684053 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38ec001fe0 con 0x7f38f413c480 2024-03-25T19:21:36.574 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.574+0000 7f38eb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:36.574 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.574+0000 7f38f1d9b640 4 set_mon_vals no callback set 2024-03-25T19:21:36.835 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.835+0000 7f38eb7fe640 1 -- 172.21.15.6:0/3058684053 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f38ec003b10 con 0x7f38f413c480 2024-03-25T19:21:36.835 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.835+0000 7f38eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:36.835 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.835+0000 7f38eb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:21:36.835 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.835+0000 7f38eb7fe640 10 monclient: dump: 2024-03-25T19:21:36.835 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:21:36.835 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:36.835 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:36.835 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:36.835 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:36.835 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:36.835 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:21:36.836 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:21:36.836 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:21:36.836 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:36.836 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.835+0000 7f38eb7fe640 1 -- 172.21.15.6:0/3058684053 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f38ec019ae0 con 0x7f38f413c480 2024-03-25T19:21:36.836 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.836+0000 7f38f2d9d640 10 monclient: get_auth_request con 0x7f38d406e010 auth_method 0 2024-03-25T19:21:36.836 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.836+0000 7f38eb7fe640 10 client.4570.objecter ms_handle_connect 0x7f38d406e010 2024-03-25T19:21:36.842 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.841+0000 7f38eb7fe640 1 -- 172.21.15.6:0/3058684053 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f38ec0538a0 con 0x7f38f413c480 2024-03-25T19:21:36.842 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.841+0000 7f38eb7fe640 10 client.4570.objecter ms_dispatch 0x7f38f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:21:36.842 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.841+0000 7f38eb7fe640 3 client.4570.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:21:36.842 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.841+0000 7f38eb7fe640 3 client.4570.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:21:36.842 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.841+0000 7f38eb7fe640 20 client.4570.objecter dump_active .. 0 homeless 2024-03-25T19:21:36.845 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.845+0000 7f38eb7fe640 1 -- 172.21.15.6:0/3058684053 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f38ec04f1e0 con 0x7f38f413c480 2024-03-25T19:21:36.845 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.845+0000 7f38eb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:36.845 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:36.845+0000 7f38eb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:21:37.077 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:37.077+0000 7f38f90c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:37.077 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:37.077+0000 7f38f90c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:37.077 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:37.077+0000 7f38f90c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:21:37.077 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:37.077+0000 7f38f90c8640 1 -- 172.21.15.6:0/3058684053 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f38f40688c0 con 0x7f38f413c480 2024-03-25T19:21:37.110 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:37.110+0000 7f38eb7fe640 1 -- 172.21.15.6:0/3058684053 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f38ec0512e0 con 0x7f38f413c480 2024-03-25T19:21:37.110 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:37.110+0000 7f38eb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:37.111 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:37.110+0000 7f38eb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:21:37.111 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:21:37.111 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:21:37.113 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:37.113+0000 7f38f90c8640 10 librados: watch_flush enter 2024-03-25T19:21:37.113 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:37.113+0000 7f38f90c8640 10 librados: watch_flush exit 2024-03-25T19:21:37.113 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:37.113+0000 7f38f90c8640 20 client.4570.objecter shutdown clearing up homeless session... 2024-03-25T19:21:37.114 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:37.113+0000 7f38f90c8640 10 client.4570.objecter successfully canceled tick 2024-03-25T19:21:37.114 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:37.113+0000 7f38f90c8640 1 -- 172.21.15.6:0/3058684053 >> v1:172.21.15.6:6800/2936652650 conn(0x7f38d406e010 legacy=0x7f38d40704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:37.114 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:37.113+0000 7f38f90c8640 10 monclient: shutdown 2024-03-25T19:21:37.114 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:37.113+0000 7f38f90c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:37.114 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:37.113+0000 7f38f90c8640 1 -- 172.21.15.6:0/3058684053 >> v1:172.21.15.192:6789/0 conn(0x7f38f413c480 legacy=0x7f38f41303c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:37.114 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:37.113+0000 7f38f90c8640 1 -- 172.21.15.6:0/3058684053 shutdown_connections 2024-03-25T19:21:37.114 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:37.113+0000 7f38f90c8640 1 -- 172.21.15.6:0/3058684053 >> 172.21.15.6:0/3058684053 conn(0x7f38f40617a0 msgr2=0x7f38f40b8690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:37.114 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:37.113+0000 7f38f90c8640 1 -- 172.21.15.6:0/3058684053 shutdown_connections 2024-03-25T19:21:37.114 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:37.113+0000 7f38f90c8640 1 -- 172.21.15.6:0/3058684053 wait complete. 2024-03-25T19:21:37.114 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:37.113+0000 7f38f90c8640 1 librados: shutdown 2024-03-25T19:21:37.126 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:21:43.932 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:21:44.048 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.048+0000 7f7614979640 10 monclient: get_monmap_and_config 2024-03-25T19:21:44.048 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.048+0000 7f7614979640 10 monclient: build_initial_monmap 2024-03-25T19:21:44.048 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.048+0000 7f7614979640 10 monclient: monmap: 2024-03-25T19:21:44.049 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:21:44.049 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:44.049 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:21:44.049642+0000 2024-03-25T19:21:44.049 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:21:44.049642+0000 2024-03-25T19:21:44.049 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:44.049 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:44.049 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:21:44.049 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:21:44.049 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:21:44.049 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:44.050 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.049+0000 7f7614979640 1 Processor -- start 2024-03-25T19:21:44.050 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.049+0000 7f7614979640 1 -- start start 2024-03-25T19:21:44.050 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.049+0000 7f7614979640 10 monclient: init 2024-03-25T19:21:44.050 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.050+0000 7f7614979640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:44.050 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.050+0000 7f7614979640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:21:44.050 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.050+0000 7f7614979640 10 monclient(hunting): picked mon.noname-a con 0x7f76100b4ff0 addr v1:172.21.15.6:6789/0 2024-03-25T19:21:44.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.050+0000 7f7614979640 10 monclient(hunting): picked mon.noname-b con 0x7f76100b4670 addr v1:172.21.15.6:6790/0 2024-03-25T19:21:44.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.050+0000 7f7614979640 10 monclient(hunting): picked mon.noname-c con 0x7f761013c480 addr v1:172.21.15.192:6789/0 2024-03-25T19:21:44.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.050+0000 7f7614979640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76100ab0e0 con 0x7f76100b4ff0 2024-03-25T19:21:44.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.050+0000 7f7614979640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76100ab3c0 con 0x7f761013c480 2024-03-25T19:21:44.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.050+0000 7f7614979640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76100a9f30 con 0x7f76100b4670 2024-03-25T19:21:44.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.050+0000 7f7614979640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:44.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.050+0000 7f7614979640 10 monclient(hunting): authenticate will time out at 6408.289942s 2024-03-25T19:21:44.061 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.061+0000 7f760e59c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f76100b4ff0 0x7f761013bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36222/0 (socket says 172.21.15.6:36222) 2024-03-25T19:21:44.061 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.061+0000 7f760e59c640 1 -- 172.21.15.6:0/610492443 learned_addr learned my addr 172.21.15.6:0/610492443 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:21:44.075 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.075+0000 7f760dd9b640 1 -- 172.21.15.6:0/610492443 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 432617322 0 0) 0x7f76100ab0e0 con 0x7f76100b4ff0 2024-03-25T19:21:44.075 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.075+0000 7f760dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:21:44.075 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.075+0000 7f760dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:44.075 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.075+0000 7f760dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:44.075 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.075+0000 7f760dd9b640 10 monclient(hunting): my global_id is 4560 2024-03-25T19:21:44.075 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.075+0000 7f760dd9b640 1 -- 172.21.15.6:0/610492443 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75fc003520 con 0x7f76100b4ff0 2024-03-25T19:21:44.079 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.079+0000 7f760dd9b640 1 -- 172.21.15.6:0/610492443 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2739177 0 0) 0x7f76100ab3c0 con 0x7f761013c480 2024-03-25T19:21:44.079 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.079+0000 7f760dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:21:44.079 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.079+0000 7f760dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:44.079 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.079+0000 7f760dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:44.079 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.079+0000 7f760dd9b640 10 monclient(hunting): my global_id is 4573 2024-03-25T19:21:44.079 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.079+0000 7f760dd9b640 1 -- 172.21.15.6:0/610492443 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76100ab0e0 con 0x7f761013c480 2024-03-25T19:21:44.081 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.081+0000 7f760dd9b640 1 -- 172.21.15.6:0/610492443 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2104496891 0 0) 0x7f76100a9f30 con 0x7f76100b4670 2024-03-25T19:21:44.081 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.081+0000 7f760dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:21:44.082 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.081+0000 7f760dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:44.082 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.081+0000 7f760dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:44.082 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.081+0000 7f760dd9b640 10 monclient(hunting): my global_id is 4568 2024-03-25T19:21:44.082 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.081+0000 7f760dd9b640 1 -- 172.21.15.6:0/610492443 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76100ab3c0 con 0x7f76100b4670 2024-03-25T19:21:44.098 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.096+0000 7f760dd9b640 1 -- 172.21.15.6:0/610492443 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1704073185 0 0) 0x7f75fc003520 con 0x7f76100b4ff0 2024-03-25T19:21:44.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.096+0000 7f760dd9b640 1 -- 172.21.15.6:0/610492443 >> v1:172.21.15.6:6790/0 conn(0x7f76100b4670 legacy=0x7f76101425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:44.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.096+0000 7f760dd9b640 1 -- 172.21.15.6:0/610492443 >> v1:172.21.15.192:6789/0 conn(0x7f761013c480 legacy=0x7f761014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:44.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.096+0000 7f760dd9b640 10 monclient: _finish_hunting 0 2024-03-25T19:21:44.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.096+0000 7f760dd9b640 1 monclient: found mon.noname-a 2024-03-25T19:21:44.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.096+0000 7f760dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:44.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.096+0000 7f760dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-03-25T19:21:44.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.096+0000 7f760dd9b640 1 -- 172.21.15.6:0/610492443 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76100a9b70 con 0x7f76100b4ff0 2024-03-25T19:21:44.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.096+0000 7f760dd9b640 10 monclient: _finish_auth 0 2024-03-25T19:21:44.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.096+0000 7f760dd9b640 10 monclient: _check_auth_tickets 2024-03-25T19:21:44.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.096+0000 7f760dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:44.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.096+0000 7f760dd9b640 1 -- 172.21.15.6:0/610492443 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f7600001690 con 0x7f76100b4ff0 2024-03-25T19:21:44.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.096+0000 7f7614979640 5 monclient: authenticate success, global_id 4560 2024-03-25T19:21:44.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.096+0000 7f7614979640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:21:44.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.096+0000 7f760dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:44.100 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.096+0000 7f760dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:21:44.100 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.096+0000 7f760dd9b640 10 monclient: dump: 2024-03-25T19:21:44.100 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:21:44.100 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:44.100 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:44.100 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:44.100 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:44.100 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:44.100 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:21:44.100 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:21:44.100 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:21:44.100 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:44.100 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.096+0000 7f7614979640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:21:44.113 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.113+0000 7f760dd9b640 1 -- 172.21.15.6:0/610492443 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76000017f0 con 0x7f76100b4ff0 2024-03-25T19:21:44.113 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.113+0000 7f760dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:44.114 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.113+0000 7f760dd9b640 1 -- 172.21.15.6:0/610492443 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f7600001c00 con 0x7f76100b4ff0 2024-03-25T19:21:44.114 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.113+0000 7f7614979640 10 monclient: get_monmap_and_config success 2024-03-25T19:21:44.114 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.113+0000 7f7614979640 4 set_mon_vals no callback set 2024-03-25T19:21:44.114 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.113+0000 7f760dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:44.114 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.113+0000 7f760dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:21:44.114 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.113+0000 7f760dd9b640 10 monclient: dump: 2024-03-25T19:21:44.114 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:21:44.114 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:44.114 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:44.114 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:44.114 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:44.114 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:44.114 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:21:44.114 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:21:44.114 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:21:44.115 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:44.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.114+0000 7f7614979640 10 monclient: shutdown 2024-03-25T19:21:44.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.114+0000 7f7614979640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:44.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.114+0000 7f7614979640 1 -- 172.21.15.6:0/610492443 >> v1:172.21.15.6:6789/0 conn(0x7f76100b4ff0 legacy=0x7f761013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:44.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.114+0000 7f7614979640 1 -- 172.21.15.6:0/610492443 shutdown_connections 2024-03-25T19:21:44.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.114+0000 7f7614979640 1 -- 172.21.15.6:0/610492443 >> 172.21.15.6:0/610492443 conn(0x7f76100617a0 msgr2=0x7f7610061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:44.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.114+0000 7f7614979640 1 -- 172.21.15.6:0/610492443 shutdown_connections 2024-03-25T19:21:44.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.114+0000 7f7614979640 1 -- 172.21.15.6:0/610492443 wait complete. 2024-03-25T19:21:44.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.115+0000 7f7614979640 10 monclient: build_initial_monmap 2024-03-25T19:21:44.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.115+0000 7f7614979640 10 monclient: monmap: 2024-03-25T19:21:44.117 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:21:44.117 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:21:44.117 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:21:44.117 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:21:44.117 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:44.117 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:21:44.117 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:21:44.117 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:21:44.118 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:21:44.118 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:44.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.115+0000 7f7614979640 1 librados: starting msgr at 2024-03-25T19:21:44.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.115+0000 7f7614979640 1 librados: starting objecter 2024-03-25T19:21:44.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.116+0000 7f7614979640 1 Processor -- start 2024-03-25T19:21:44.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.116+0000 7f7614979640 1 -- start start 2024-03-25T19:21:44.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.116+0000 7f7614979640 1 librados: setting wanted keys 2024-03-25T19:21:44.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.116+0000 7f7614979640 1 librados: calling monclient init 2024-03-25T19:21:44.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.116+0000 7f7614979640 10 monclient: init 2024-03-25T19:21:44.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.116+0000 7f7614979640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:44.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.116+0000 7f7614979640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:21:44.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.116+0000 7f7614979640 10 monclient(hunting): picked mon.noname-c con 0x7f76100b4670 addr v1:172.21.15.6:6790/0 2024-03-25T19:21:44.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.116+0000 7f7614979640 10 monclient(hunting): picked mon.noname-b con 0x7f76100b4ff0 addr v1:172.21.15.192:6789/0 2024-03-25T19:21:44.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.116+0000 7f7614979640 10 monclient(hunting): picked mon.noname-a con 0x7f761013c480 addr v1:172.21.15.6:6789/0 2024-03-25T19:21:44.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.116+0000 7f7614979640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7610135ea0 con 0x7f761013c480 2024-03-25T19:21:44.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.116+0000 7f7614979640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76100ac3d0 con 0x7f76100b4ff0 2024-03-25T19:21:44.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.117+0000 7f7614979640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76100ac6b0 con 0x7f76100b4670 2024-03-25T19:21:44.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.117+0000 7f7614979640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:44.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.117+0000 7f7614979640 10 monclient(hunting): authenticate will time out at 6408.356815s 2024-03-25T19:21:44.125 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.125+0000 7f760ed9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f761013c480 0x7f76101362d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36268/0 (socket says 172.21.15.6:36268) 2024-03-25T19:21:44.125 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.125+0000 7f760ed9d640 1 -- 172.21.15.6:0/2770683536 learned_addr learned my addr 172.21.15.6:0/2770683536 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:21:44.128 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.129+0000 7f76077fe640 10 client.?.objecter ms_handle_connect 0x7f76100b4ff0 2024-03-25T19:21:44.128 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.129+0000 7f76077fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:44.130 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.130+0000 7f76077fe640 10 client.?.objecter ms_handle_connect 0x7f761013c480 2024-03-25T19:21:44.130 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.130+0000 7f76077fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:44.132 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.132+0000 7f76077fe640 10 client.?.objecter ms_handle_connect 0x7f76100b4670 2024-03-25T19:21:44.132 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.132+0000 7f76077fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:44.138 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.138+0000 7f76077fe640 1 -- 172.21.15.6:0/2770683536 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2986075117 0 0) 0x7f76100ac3d0 con 0x7f76100b4ff0 2024-03-25T19:21:44.138 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.138+0000 7f76077fe640 10 monclient(hunting): _negotiate 2024-03-25T19:21:44.139 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.138+0000 7f76077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:44.139 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.138+0000 7f76077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:44.139 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.138+0000 7f76077fe640 10 monclient(hunting): my global_id is 4576 2024-03-25T19:21:44.139 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.138+0000 7f76077fe640 1 -- 172.21.15.6:0/2770683536 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75f0004eb0 con 0x7f76100b4ff0 2024-03-25T19:21:44.142 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.142+0000 7f76077fe640 1 -- 172.21.15.6:0/2770683536 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2719324712 0 0) 0x7f7610135ea0 con 0x7f761013c480 2024-03-25T19:21:44.142 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.142+0000 7f76077fe640 10 monclient(hunting): _negotiate 2024-03-25T19:21:44.142 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.142+0000 7f76077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:44.142 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.142+0000 7f76077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:44.142 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.142+0000 7f76077fe640 10 monclient(hunting): my global_id is 4563 2024-03-25T19:21:44.142 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.142+0000 7f76077fe640 1 -- 172.21.15.6:0/2770683536 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76100ac3d0 con 0x7f761013c480 2024-03-25T19:21:44.151 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.151+0000 7f76077fe640 1 -- 172.21.15.6:0/2770683536 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2963620589 0 0) 0x7f76100ac3d0 con 0x7f761013c480 2024-03-25T19:21:44.151 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.151+0000 7f76077fe640 1 -- 172.21.15.6:0/2770683536 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7610135ea0 con 0x7f761013c480 2024-03-25T19:21:44.152 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.151+0000 7f76077fe640 1 -- 172.21.15.6:0/2770683536 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f7608001c10 con 0x7f761013c480 2024-03-25T19:21:44.152 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.151+0000 7f76077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:44.152 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.151+0000 7f76077fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:21:44.152 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.151+0000 7f76077fe640 10 monclient(hunting): dump: 2024-03-25T19:21:44.152 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:21:44.152 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:44.152 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:44.152 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:44.152 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:44.152 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:44.152 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:21:44.152 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:21:44.152 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:21:44.152 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:44.153 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.151+0000 7f76077fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:21:44.153 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.153+0000 7f76077fe640 1 -- 172.21.15.6:0/2770683536 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2424323697 0 0) 0x7f76100ac6b0 con 0x7f76100b4670 2024-03-25T19:21:44.153 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.153+0000 7f76077fe640 10 monclient(hunting): _negotiate 2024-03-25T19:21:44.153 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.153+0000 7f76077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:44.153 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.153+0000 7f76077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:44.153 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.153+0000 7f76077fe640 10 monclient(hunting): my global_id is 4571 2024-03-25T19:21:44.153 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.153+0000 7f76077fe640 1 -- 172.21.15.6:0/2770683536 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75f00057f0 con 0x7f76100b4670 2024-03-25T19:21:44.155 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.155+0000 7f76077fe640 1 -- 172.21.15.6:0/2770683536 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3113093858 0 0) 0x7f75f0004eb0 con 0x7f76100b4ff0 2024-03-25T19:21:44.155 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.155+0000 7f76077fe640 1 -- 172.21.15.6:0/2770683536 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76100ac6b0 con 0x7f76100b4ff0 2024-03-25T19:21:44.156 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.155+0000 7f76077fe640 1 -- 172.21.15.6:0/2770683536 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f75f80018f0 con 0x7f76100b4ff0 2024-03-25T19:21:44.156 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.155+0000 7f76077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:44.156 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.155+0000 7f76077fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:21:44.156 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.155+0000 7f76077fe640 10 monclient(hunting): dump: 2024-03-25T19:21:44.156 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:21:44.156 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:44.156 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:44.156 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:44.156 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:44.156 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:44.156 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:21:44.156 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:21:44.156 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:21:44.156 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:44.161 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.161+0000 7f76077fe640 1 -- 172.21.15.6:0/2770683536 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2636409285 0 0) 0x7f7610135ea0 con 0x7f761013c480 2024-03-25T19:21:44.162 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.161+0000 7f76077fe640 1 -- 172.21.15.6:0/2770683536 >> v1:172.21.15.6:6790/0 conn(0x7f76100b4670 legacy=0x7f761013b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:44.162 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.161+0000 7f76077fe640 1 -- 172.21.15.6:0/2770683536 >> v1:172.21.15.192:6789/0 conn(0x7f76100b4ff0 legacy=0x7f76100b0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:44.162 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.161+0000 7f76077fe640 10 monclient: _finish_hunting 0 2024-03-25T19:21:44.162 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.161+0000 7f76077fe640 1 monclient: found mon.a 2024-03-25T19:21:44.162 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.161+0000 7f76077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:44.162 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.161+0000 7f76077fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:21:44.162 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.161+0000 7f76077fe640 1 -- 172.21.15.6:0/2770683536 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f761013c110 con 0x7f761013c480 2024-03-25T19:21:44.162 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.161+0000 7f76077fe640 10 monclient: _finish_auth 0 2024-03-25T19:21:44.162 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.161+0000 7f76077fe640 10 monclient: _check_auth_tickets 2024-03-25T19:21:44.162 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.161+0000 7f76077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:44.162 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.161+0000 7f7614979640 5 monclient: authenticate success, global_id 4563 2024-03-25T19:21:44.162 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.161+0000 7f7614979640 10 monclient: _renew_subs 2024-03-25T19:21:44.162 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.161+0000 7f7614979640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:21:44.163 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.161+0000 7f7614979640 1 -- 172.21.15.6:0/2770683536 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f761013ae10 con 0x7f761013c480 2024-03-25T19:21:44.163 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.161+0000 7f7614979640 10 client.4563.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:21:44.163 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.161+0000 7f7614979640 10 monclient: _renew_subs 2024-03-25T19:21:44.163 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.161+0000 7f7614979640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:21:44.163 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.161+0000 7f7614979640 1 -- 172.21.15.6:0/2770683536 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f76100b1550 con 0x7f761013c480 2024-03-25T19:21:44.163 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.161+0000 7f7614979640 1 librados: init done 2024-03-25T19:21:44.163 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.162+0000 7f7614979640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:44.164 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.162+0000 7f7614979640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:44.164 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.162+0000 7f7614979640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:21:44.164 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.162+0000 7f7614979640 1 -- 172.21.15.6:0/2770683536 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7610146790 con 0x7f761013c480 2024-03-25T19:21:44.491 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.491+0000 7f76077fe640 1 -- 172.21.15.6:0/2770683536 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7608003970 con 0x7f761013c480 2024-03-25T19:21:44.492 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.491+0000 7f76077fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:44.492 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.491+0000 7f76077fe640 1 -- 172.21.15.6:0/2770683536 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f7608003d00 con 0x7f761013c480 2024-03-25T19:21:44.492 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.491+0000 7f76077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:44.492 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.491+0000 7f76077fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:21:44.492 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.491+0000 7f76077fe640 10 monclient: dump: 2024-03-25T19:21:44.492 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:21:44.492 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:44.492 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:44.492 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:44.492 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:44.492 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:44.492 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:21:44.493 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:21:44.493 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:21:44.493 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:44.493 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.491+0000 7f760d59a640 4 set_mon_vals no callback set 2024-03-25T19:21:44.493 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.492+0000 7f76077fe640 1 -- 172.21.15.6:0/2770683536 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f7608003f20 con 0x7f761013c480 2024-03-25T19:21:44.493 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.492+0000 7f760e59c640 10 monclient: get_auth_request con 0x7f75f006e170 auth_method 0 2024-03-25T19:21:44.493 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.492+0000 7f76077fe640 10 client.4563.objecter ms_handle_connect 0x7f75f006e170 2024-03-25T19:21:44.498 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.498+0000 7f76077fe640 1 -- 172.21.15.6:0/2770683536 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f76080549f0 con 0x7f761013c480 2024-03-25T19:21:44.498 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.498+0000 7f76077fe640 10 client.4563.objecter ms_dispatch 0x7f7610000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:21:44.498 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.498+0000 7f76077fe640 3 client.4563.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:21:44.498 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.498+0000 7f76077fe640 3 client.4563.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:21:44.498 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.498+0000 7f76077fe640 20 client.4563.objecter dump_active .. 0 homeless 2024-03-25T19:21:44.498 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.498+0000 7f76077fe640 1 -- 172.21.15.6:0/2770683536 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7608050500 con 0x7f761013c480 2024-03-25T19:21:44.498 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.498+0000 7f76077fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:44.498 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.498+0000 7f76077fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:21:44.757 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.757+0000 7f7614979640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:44.757 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.757+0000 7f7614979640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:44.757 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.757+0000 7f7614979640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:21:44.757 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.757+0000 7f7614979640 1 -- 172.21.15.6:0/2770683536 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f761014f100 con 0x7f761013c480 2024-03-25T19:21:44.802 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.802+0000 7f76077fe640 1 -- 172.21.15.6:0/2770683536 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f7608052430 con 0x7f761013c480 2024-03-25T19:21:44.802 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.802+0000 7f76077fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:44.802 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.802+0000 7f76077fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:21:44.802 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:21:44.803 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:21:44.805 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.804+0000 7f7614979640 10 librados: watch_flush enter 2024-03-25T19:21:44.805 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.804+0000 7f7614979640 10 librados: watch_flush exit 2024-03-25T19:21:44.805 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.805+0000 7f7614979640 20 client.4563.objecter shutdown clearing up homeless session... 2024-03-25T19:21:44.805 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.805+0000 7f7614979640 10 client.4563.objecter successfully canceled tick 2024-03-25T19:21:44.805 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.805+0000 7f7614979640 1 -- 172.21.15.6:0/2770683536 >> v1:172.21.15.6:6800/2936652650 conn(0x7f75f006e170 legacy=0x7f75f0070630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:44.806 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.805+0000 7f7614979640 10 monclient: shutdown 2024-03-25T19:21:44.806 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.805+0000 7f7614979640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:44.806 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.805+0000 7f7614979640 1 -- 172.21.15.6:0/2770683536 >> v1:172.21.15.6:6789/0 conn(0x7f761013c480 legacy=0x7f76101362d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:44.806 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.805+0000 7f7614979640 1 -- 172.21.15.6:0/2770683536 shutdown_connections 2024-03-25T19:21:44.806 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.805+0000 7f7614979640 1 -- 172.21.15.6:0/2770683536 >> 172.21.15.6:0/2770683536 conn(0x7f76100617a0 msgr2=0x7f76101407a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:44.806 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.805+0000 7f7614979640 1 -- 172.21.15.6:0/2770683536 shutdown_connections 2024-03-25T19:21:44.806 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.805+0000 7f7614979640 1 -- 172.21.15.6:0/2770683536 wait complete. 2024-03-25T19:21:44.806 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:44.805+0000 7f7614979640 1 librados: shutdown 2024-03-25T19:21:44.816 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:21:48.896 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:08:06.243 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:48.944 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:08:06.291 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:48.981 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:08:06.328 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:48.996 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:08:06.314 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:49.004 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:08:06.322 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:49.014 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:08:06.361 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:49.022 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:08:06.369 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:49.026 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:08:06.344 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:49.046 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:08:06.393 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:51.623 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:21:51.747 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.746+0000 7fcbe4663640 10 monclient: get_monmap_and_config 2024-03-25T19:21:51.747 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.746+0000 7fcbe4663640 10 monclient: build_initial_monmap 2024-03-25T19:21:51.747 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.746+0000 7fcbe4663640 10 monclient: monmap: 2024-03-25T19:21:51.747 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:21:51.747 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:51.747 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:21:51.747427+0000 2024-03-25T19:21:51.747 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:21:51.747427+0000 2024-03-25T19:21:51.747 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:51.748 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:51.748 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:21:51.748 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:21:51.748 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:21:51.748 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:51.748 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.747+0000 7fcbe4663640 1 Processor -- start 2024-03-25T19:21:51.748 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.747+0000 7fcbe4663640 1 -- start start 2024-03-25T19:21:51.748 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.747+0000 7fcbe4663640 10 monclient: init 2024-03-25T19:21:51.749 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.748+0000 7fcbe4663640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:51.749 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.748+0000 7fcbe4663640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:21:51.749 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.748+0000 7fcbe4663640 10 monclient(hunting): picked mon.noname-b con 0x7fcbdc0b4ff0 addr v1:172.21.15.6:6790/0 2024-03-25T19:21:51.749 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.748+0000 7fcbe4663640 10 monclient(hunting): picked mon.noname-c con 0x7fcbdc0b4670 addr v1:172.21.15.192:6789/0 2024-03-25T19:21:51.749 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.748+0000 7fcbe4663640 10 monclient(hunting): picked mon.noname-a con 0x7fcbdc13c480 addr v1:172.21.15.6:6789/0 2024-03-25T19:21:51.750 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.748+0000 7fcbe4663640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbdc0ab0e0 con 0x7fcbdc13c480 2024-03-25T19:21:51.750 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.748+0000 7fcbe4663640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbdc0ab3c0 con 0x7fcbdc0b4670 2024-03-25T19:21:51.750 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.748+0000 7fcbe4663640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbdc0a9f30 con 0x7fcbdc0b4ff0 2024-03-25T19:21:51.750 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.748+0000 7fcbe4663640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:51.750 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.748+0000 7fcbe4663640 10 monclient(hunting): authenticate will time out at 6415.988048s 2024-03-25T19:21:51.758 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.758+0000 7fcbe1bfe640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fcbdc0b4670 0x7fcbdc1425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:36234/0 (socket says 172.21.15.6:36234) 2024-03-25T19:21:51.759 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.758+0000 7fcbe1bfe640 1 -- 172.21.15.6:0/428035949 learned_addr learned my addr 172.21.15.6:0/428035949 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:21:51.769 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.769+0000 7fcbe13fd640 1 -- 172.21.15.6:0/428035949 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2855244131 0 0) 0x7fcbdc0a9f30 con 0x7fcbdc0b4ff0 2024-03-25T19:21:51.769 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.769+0000 7fcbe13fd640 10 monclient(hunting): _negotiate 2024-03-25T19:21:51.769 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.769+0000 7fcbe13fd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:51.769 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.769+0000 7fcbe13fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:51.769 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.769+0000 7fcbe13fd640 10 monclient(hunting): my global_id is 4574 2024-03-25T19:21:51.769 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.769+0000 7fcbe13fd640 1 -- 172.21.15.6:0/428035949 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbc8003520 con 0x7fcbdc0b4ff0 2024-03-25T19:21:51.774 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.774+0000 7fcbe13fd640 1 -- 172.21.15.6:0/428035949 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 914411466 0 0) 0x7fcbdc0ab3c0 con 0x7fcbdc0b4670 2024-03-25T19:21:51.774 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.774+0000 7fcbe13fd640 10 monclient(hunting): _negotiate 2024-03-25T19:21:51.774 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.774+0000 7fcbe13fd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:51.774 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.774+0000 7fcbe13fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:51.774 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.774+0000 7fcbe13fd640 10 monclient(hunting): my global_id is 4579 2024-03-25T19:21:51.774 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.774+0000 7fcbe13fd640 1 -- 172.21.15.6:0/428035949 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbdc0a9f30 con 0x7fcbdc0b4670 2024-03-25T19:21:51.782 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.782+0000 7fcbe13fd640 1 -- 172.21.15.6:0/428035949 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3605740138 0 0) 0x7fcbdc0a9f30 con 0x7fcbdc0b4670 2024-03-25T19:21:51.782 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.782+0000 7fcbe13fd640 1 -- 172.21.15.6:0/428035949 >> v1:172.21.15.6:6790/0 conn(0x7fcbdc0b4ff0 legacy=0x7fcbdc13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:51.782 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.782+0000 7fcbe13fd640 1 -- 172.21.15.6:0/428035949 >> v1:172.21.15.6:6789/0 conn(0x7fcbdc13c480 legacy=0x7fcbdc14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:51.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.782+0000 7fcbe13fd640 10 monclient: _finish_hunting 0 2024-03-25T19:21:51.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.782+0000 7fcbe13fd640 1 monclient: found mon.noname-c 2024-03-25T19:21:51.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.782+0000 7fcbe13fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:51.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.782+0000 7fcbe13fd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:21:51.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.782+0000 7fcbe13fd640 1 -- 172.21.15.6:0/428035949 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbdc0a9b70 con 0x7fcbdc0b4670 2024-03-25T19:21:51.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.782+0000 7fcbe13fd640 10 monclient: _finish_auth 0 2024-03-25T19:21:51.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.782+0000 7fcbe13fd640 10 monclient: _check_auth_tickets 2024-03-25T19:21:51.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.782+0000 7fcbe13fd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:51.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.782+0000 7fcbe13fd640 1 -- 172.21.15.6:0/428035949 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fcbcc001670 con 0x7fcbdc0b4670 2024-03-25T19:21:51.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.782+0000 7fcbe13fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:51.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.782+0000 7fcbe13fd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:21:51.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.782+0000 7fcbe13fd640 10 monclient: dump: 2024-03-25T19:21:51.783 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:21:51.783 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:51.783 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:51.783 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:51.784 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:51.784 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:51.784 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:21:51.784 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:21:51.784 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:21:51.784 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:51.784 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.782+0000 7fcbe4663640 5 monclient: authenticate success, global_id 4579 2024-03-25T19:21:51.784 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.782+0000 7fcbe4663640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:21:51.802 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.802+0000 7fcbe13fd640 1 -- 172.21.15.6:0/428035949 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcbcc0017d0 con 0x7fcbdc0b4670 2024-03-25T19:21:51.802 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.802+0000 7fcbe13fd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:51.802 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.802+0000 7fcbe13fd640 1 -- 172.21.15.6:0/428035949 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fcbcc001ba0 con 0x7fcbdc0b4670 2024-03-25T19:21:51.802 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.802+0000 7fcbe4663640 10 monclient: get_monmap_and_config success 2024-03-25T19:21:51.802 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.802+0000 7fcbe4663640 4 set_mon_vals no callback set 2024-03-25T19:21:51.802 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.802+0000 7fcbe13fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:51.802 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.802+0000 7fcbe13fd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:21:51.802 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.802+0000 7fcbe13fd640 10 monclient: dump: 2024-03-25T19:21:51.802 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:21:51.802 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:51.802 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:51.803 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:51.803 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:51.803 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:51.803 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:21:51.803 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:21:51.803 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:21:51.803 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:51.805 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.802+0000 7fcbe4663640 10 monclient: shutdown 2024-03-25T19:21:51.805 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.802+0000 7fcbe4663640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:51.805 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.802+0000 7fcbe4663640 1 -- 172.21.15.6:0/428035949 >> v1:172.21.15.192:6789/0 conn(0x7fcbdc0b4670 legacy=0x7fcbdc1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:51.805 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.803+0000 7fcbe4663640 1 -- 172.21.15.6:0/428035949 shutdown_connections 2024-03-25T19:21:51.806 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.803+0000 7fcbe4663640 1 -- 172.21.15.6:0/428035949 >> 172.21.15.6:0/428035949 conn(0x7fcbdc0617a0 msgr2=0x7fcbdc061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:51.806 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.803+0000 7fcbe4663640 1 -- 172.21.15.6:0/428035949 shutdown_connections 2024-03-25T19:21:51.806 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.803+0000 7fcbe4663640 1 -- 172.21.15.6:0/428035949 wait complete. 2024-03-25T19:21:51.806 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.803+0000 7fcbe4663640 10 monclient: build_initial_monmap 2024-03-25T19:21:51.806 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.803+0000 7fcbe4663640 10 monclient: monmap: 2024-03-25T19:21:51.806 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:21:51.806 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:21:51.806 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:21:51.806 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:21:51.806 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:51.806 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:21:51.806 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:21:51.806 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:21:51.806 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:21:51.807 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:51.807 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.804+0000 7fcbe4663640 1 librados: starting msgr at 2024-03-25T19:21:51.807 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.804+0000 7fcbe4663640 1 librados: starting objecter 2024-03-25T19:21:51.807 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.804+0000 7fcbe4663640 1 Processor -- start 2024-03-25T19:21:51.807 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.804+0000 7fcbe4663640 1 -- start start 2024-03-25T19:21:51.807 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.804+0000 7fcbe4663640 1 librados: setting wanted keys 2024-03-25T19:21:51.807 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.804+0000 7fcbe4663640 1 librados: calling monclient init 2024-03-25T19:21:51.807 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.804+0000 7fcbe4663640 10 monclient: init 2024-03-25T19:21:51.807 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.804+0000 7fcbe4663640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:51.807 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.804+0000 7fcbe4663640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:21:51.807 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.804+0000 7fcbe4663640 10 monclient(hunting): picked mon.noname-b con 0x7fcbdc0b4670 addr v1:172.21.15.192:6789/0 2024-03-25T19:21:51.807 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.804+0000 7fcbe4663640 10 monclient(hunting): picked mon.noname-c con 0x7fcbdc0b4ff0 addr v1:172.21.15.6:6790/0 2024-03-25T19:21:51.807 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.804+0000 7fcbe4663640 10 monclient(hunting): picked mon.noname-a con 0x7fcbdc13c480 addr v1:172.21.15.6:6789/0 2024-03-25T19:21:51.807 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.805+0000 7fcbe4663640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbdc06e920 con 0x7fcbdc13c480 2024-03-25T19:21:51.808 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.805+0000 7fcbe4663640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbdc06eb70 con 0x7fcbdc0b4670 2024-03-25T19:21:51.808 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.805+0000 7fcbe4663640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbdc13ead0 con 0x7fcbdc0b4ff0 2024-03-25T19:21:51.808 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.805+0000 7fcbe4663640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:51.808 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.805+0000 7fcbe4663640 10 monclient(hunting): authenticate will time out at 6416.044821s 2024-03-25T19:21:51.810 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.810+0000 7fcbe2c00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fcbdc13c480 0x7fcbdc1e6790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53832/0 (socket says 172.21.15.6:53832) 2024-03-25T19:21:51.810 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.810+0000 7fcbe2c00640 1 -- 172.21.15.6:0/1382960503 learned_addr learned my addr 172.21.15.6:0/1382960503 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:21:51.811 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.811+0000 7fcbd2ffd640 10 client.?.objecter ms_handle_connect 0x7fcbdc13c480 2024-03-25T19:21:51.811 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.811+0000 7fcbd2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:51.811 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.812+0000 7fcbd2ffd640 10 client.?.objecter ms_handle_connect 0x7fcbdc0b4ff0 2024-03-25T19:21:51.812 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.812+0000 7fcbd2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:51.815 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.816+0000 7fcbd2ffd640 10 client.?.objecter ms_handle_connect 0x7fcbdc0b4670 2024-03-25T19:21:51.815 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.816+0000 7fcbd2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:51.826 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.826+0000 7fcbd2ffd640 1 -- 172.21.15.6:0/1382960503 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1120075116 0 0) 0x7fcbdc06eb70 con 0x7fcbdc0b4670 2024-03-25T19:21:51.826 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.826+0000 7fcbd2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:21:51.826 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.826+0000 7fcbd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:51.826 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.826+0000 7fcbd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:51.826 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.826+0000 7fcbd2ffd640 10 monclient(hunting): my global_id is 4582 2024-03-25T19:21:51.826 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.826+0000 7fcbd2ffd640 1 -- 172.21.15.6:0/1382960503 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbc0004eb0 con 0x7fcbdc0b4670 2024-03-25T19:21:51.826 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.826+0000 7fcbd2ffd640 1 -- 172.21.15.6:0/1382960503 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 859948447 0 0) 0x7fcbdc06e920 con 0x7fcbdc13c480 2024-03-25T19:21:51.826 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.826+0000 7fcbd2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:21:51.827 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.826+0000 7fcbd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:51.827 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.826+0000 7fcbd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:51.827 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.826+0000 7fcbd2ffd640 10 monclient(hunting): my global_id is 4569 2024-03-25T19:21:51.827 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.826+0000 7fcbd2ffd640 1 -- 172.21.15.6:0/1382960503 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbdc06eb70 con 0x7fcbdc13c480 2024-03-25T19:21:51.827 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.827+0000 7fcbd2ffd640 1 -- 172.21.15.6:0/1382960503 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3373473703 0 0) 0x7fcbdc13ead0 con 0x7fcbdc0b4ff0 2024-03-25T19:21:51.827 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.827+0000 7fcbd2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:21:51.827 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.827+0000 7fcbd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:51.827 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.827+0000 7fcbd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:51.828 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.827+0000 7fcbd2ffd640 10 monclient(hunting): my global_id is 4577 2024-03-25T19:21:51.828 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.827+0000 7fcbd2ffd640 1 -- 172.21.15.6:0/1382960503 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbdc06e920 con 0x7fcbdc0b4ff0 2024-03-25T19:21:51.836 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.836+0000 7fcbd2ffd640 1 -- 172.21.15.6:0/1382960503 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2320119242 0 0) 0x7fcbdc06eb70 con 0x7fcbdc13c480 2024-03-25T19:21:51.837 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.836+0000 7fcbd2ffd640 1 -- 172.21.15.6:0/1382960503 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcbdc13ead0 con 0x7fcbdc13c480 2024-03-25T19:21:51.837 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.836+0000 7fcbd2ffd640 1 -- 172.21.15.6:0/1382960503 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fcbd4001c10 con 0x7fcbdc13c480 2024-03-25T19:21:51.837 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.836+0000 7fcbd2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:51.837 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.836+0000 7fcbd2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:21:51.837 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.836+0000 7fcbd2ffd640 10 monclient(hunting): dump: 2024-03-25T19:21:51.837 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:21:51.837 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:51.837 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:51.837 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:51.837 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:51.837 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:51.838 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:21:51.838 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:21:51.838 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:21:51.838 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:51.838 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.836+0000 7fcbd2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:21:51.838 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.837+0000 7fcbd2ffd640 1 -- 172.21.15.6:0/1382960503 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4000990261 0 0) 0x7fcbc0004eb0 con 0x7fcbdc0b4670 2024-03-25T19:21:51.838 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.837+0000 7fcbd2ffd640 1 -- 172.21.15.6:0/1382960503 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcbdc06eb70 con 0x7fcbdc0b4670 2024-03-25T19:21:51.838 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.837+0000 7fcbd2ffd640 1 -- 172.21.15.6:0/1382960503 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fcbd8001950 con 0x7fcbdc0b4670 2024-03-25T19:21:51.838 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.837+0000 7fcbd2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:51.838 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.837+0000 7fcbd2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:21:51.838 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.837+0000 7fcbd2ffd640 10 monclient(hunting): dump: 2024-03-25T19:21:51.838 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:21:51.839 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:51.839 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:51.839 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:51.839 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:51.839 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:51.839 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:21:51.839 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:21:51.839 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:21:51.839 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:51.840 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.840+0000 7fcbd2ffd640 1 -- 172.21.15.6:0/1382960503 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4176576295 0 0) 0x7fcbdc13ead0 con 0x7fcbdc13c480 2024-03-25T19:21:51.840 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.840+0000 7fcbd2ffd640 1 -- 172.21.15.6:0/1382960503 >> v1:172.21.15.6:6790/0 conn(0x7fcbdc0b4ff0 legacy=0x7fcbdc13eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:51.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.840+0000 7fcbd2ffd640 1 -- 172.21.15.6:0/1382960503 >> v1:172.21.15.192:6789/0 conn(0x7fcbdc0b4670 legacy=0x7fcbdc145680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:51.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.840+0000 7fcbd2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:21:51.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.840+0000 7fcbd2ffd640 1 monclient: found mon.a 2024-03-25T19:21:51.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.840+0000 7fcbd2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:51.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.840+0000 7fcbd2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:21:51.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.840+0000 7fcbd2ffd640 1 -- 172.21.15.6:0/1382960503 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbdc13f700 con 0x7fcbdc13c480 2024-03-25T19:21:51.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.840+0000 7fcbd2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:21:51.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.840+0000 7fcbd2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:21:51.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.840+0000 7fcbd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:51.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.840+0000 7fcbe4663640 5 monclient: authenticate success, global_id 4569 2024-03-25T19:21:51.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.840+0000 7fcbe4663640 10 monclient: _renew_subs 2024-03-25T19:21:51.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.840+0000 7fcbe4663640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:21:51.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.840+0000 7fcbe4663640 1 -- 172.21.15.6:0/1382960503 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbdc13e430 con 0x7fcbdc13c480 2024-03-25T19:21:51.841 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.840+0000 7fcbe4663640 10 client.4569.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:21:51.842 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.840+0000 7fcbe4663640 10 monclient: _renew_subs 2024-03-25T19:21:51.842 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.840+0000 7fcbe4663640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:21:51.842 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.840+0000 7fcbe4663640 1 -- 172.21.15.6:0/1382960503 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbdc13c8f0 con 0x7fcbdc13c480 2024-03-25T19:21:51.842 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.840+0000 7fcbe4663640 1 librados: init done 2024-03-25T19:21:51.842 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.841+0000 7fcbe4663640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:51.842 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.841+0000 7fcbe4663640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:51.842 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.841+0000 7fcbe4663640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:21:51.843 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:51.841+0000 7fcbe4663640 1 -- 172.21.15.6:0/1382960503 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbdc13ec60 con 0x7fcbdc13c480 2024-03-25T19:21:52.088 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:52.088+0000 7fcbd2ffd640 1 -- 172.21.15.6:0/1382960503 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcbd4003990 con 0x7fcbdc13c480 2024-03-25T19:21:52.088 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:52.088+0000 7fcbd2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:52.088 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:52.088+0000 7fcbe0bfc640 4 set_mon_vals no callback set 2024-03-25T19:21:52.168 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:52.168+0000 7fcbd2ffd640 1 -- 172.21.15.6:0/1382960503 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fcbd4002100 con 0x7fcbdc13c480 2024-03-25T19:21:52.168 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:52.168+0000 7fcbd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:52.168 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:52.168+0000 7fcbd2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:21:52.168 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:52.168+0000 7fcbd2ffd640 10 monclient: dump: 2024-03-25T19:21:52.169 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:21:52.169 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:52.169 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:52.169 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:52.169 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:52.169 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:52.169 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:21:52.169 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:21:52.169 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:21:52.169 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:52.170 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:52.169+0000 7fcbd2ffd640 1 -- 172.21.15.6:0/1382960503 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7fcbd4003d60 con 0x7fcbdc13c480 2024-03-25T19:21:52.170 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:52.170+0000 7fcbe23ff640 10 monclient: get_auth_request con 0x7fcbc006df20 auth_method 0 2024-03-25T19:21:52.170 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:52.170+0000 7fcbd2ffd640 10 client.4569.objecter ms_handle_connect 0x7fcbc006df20 2024-03-25T19:21:52.175 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:52.175+0000 7fcbd2ffd640 1 -- 172.21.15.6:0/1382960503 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7fcbd4054b00 con 0x7fcbdc13c480 2024-03-25T19:21:52.175 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:52.175+0000 7fcbd2ffd640 10 client.4569.objecter ms_dispatch 0x7fcbdc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:21:52.175 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:52.175+0000 7fcbd2ffd640 3 client.4569.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:21:52.175 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:52.175+0000 7fcbd2ffd640 3 client.4569.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:21:52.175 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:52.175+0000 7fcbd2ffd640 20 client.4569.objecter dump_active .. 0 homeless 2024-03-25T19:21:52.175 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:52.175+0000 7fcbd2ffd640 1 -- 172.21.15.6:0/1382960503 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcbd4050440 con 0x7fcbdc13c480 2024-03-25T19:21:52.176 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:52.175+0000 7fcbd2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:52.176 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:52.175+0000 7fcbd2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:21:52.410 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:52.410+0000 7fcbe4663640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:52.410 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:52.410+0000 7fcbe4663640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:52.410 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:52.410+0000 7fcbe4663640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:21:52.410 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:52.410+0000 7fcbe4663640 1 -- 172.21.15.6:0/1382960503 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcbdc06dbd0 con 0x7fcbdc13c480 2024-03-25T19:21:52.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:52.444+0000 7fcbd2ffd640 1 -- 172.21.15.6:0/1382960503 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7fcbd4052540 con 0x7fcbdc13c480 2024-03-25T19:21:52.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:52.444+0000 7fcbd2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:52.445 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:52.444+0000 7fcbd2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:21:52.445 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:21:52.445 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:21:52.447 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:52.446+0000 7fcbe4663640 10 librados: watch_flush enter 2024-03-25T19:21:52.447 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:52.446+0000 7fcbe4663640 10 librados: watch_flush exit 2024-03-25T19:21:52.447 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:52.446+0000 7fcbe4663640 20 client.4569.objecter shutdown clearing up homeless session... 2024-03-25T19:21:52.447 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:52.446+0000 7fcbe4663640 10 client.4569.objecter successfully canceled tick 2024-03-25T19:21:52.447 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:52.447+0000 7fcbe4663640 1 -- 172.21.15.6:0/1382960503 >> v1:172.21.15.6:6800/2936652650 conn(0x7fcbc006df20 legacy=0x7fcbc00703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:52.448 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:52.447+0000 7fcbe4663640 10 monclient: shutdown 2024-03-25T19:21:52.448 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:52.447+0000 7fcbe4663640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:52.448 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:52.447+0000 7fcbe4663640 1 -- 172.21.15.6:0/1382960503 >> v1:172.21.15.6:6789/0 conn(0x7fcbdc13c480 legacy=0x7fcbdc1e6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:52.448 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:52.447+0000 7fcbe4663640 1 -- 172.21.15.6:0/1382960503 shutdown_connections 2024-03-25T19:21:52.448 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:52.447+0000 7fcbe4663640 1 -- 172.21.15.6:0/1382960503 >> 172.21.15.6:0/1382960503 conn(0x7fcbdc0617a0 msgr2=0x7fcbdc13bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:52.448 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:52.447+0000 7fcbe4663640 1 -- 172.21.15.6:0/1382960503 shutdown_connections 2024-03-25T19:21:52.448 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:52.447+0000 7fcbe4663640 1 -- 172.21.15.6:0/1382960503 wait complete. 2024-03-25T19:21:52.448 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:52.447+0000 7fcbe4663640 1 librados: shutdown 2024-03-25T19:21:52.459 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:21:59.265 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:21:59.384 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.384+0000 7fabad16d640 10 monclient: get_monmap_and_config 2024-03-25T19:21:59.385 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.384+0000 7fabad16d640 10 monclient: build_initial_monmap 2024-03-25T19:21:59.385 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.384+0000 7fabad16d640 10 monclient: monmap: 2024-03-25T19:21:59.385 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:21:59.385 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:59.385 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:21:59.385502+0000 2024-03-25T19:21:59.385 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:21:59.385502+0000 2024-03-25T19:21:59.385 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:59.385 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:59.385 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:21:59.385 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:21:59.385 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:21:59.385 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:59.386 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.385+0000 7fabad16d640 1 Processor -- start 2024-03-25T19:21:59.387 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.385+0000 7fabad16d640 1 -- start start 2024-03-25T19:21:59.387 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.385+0000 7fabad16d640 10 monclient: init 2024-03-25T19:21:59.387 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.386+0000 7fabad16d640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:59.387 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.386+0000 7fabad16d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:21:59.387 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.386+0000 7fabad16d640 10 monclient(hunting): picked mon.noname-b con 0x7faba80b4e70 addr v1:172.21.15.6:6790/0 2024-03-25T19:21:59.387 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.386+0000 7fabad16d640 10 monclient(hunting): picked mon.noname-a con 0x7faba80b44b0 addr v1:172.21.15.6:6789/0 2024-03-25T19:21:59.387 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.386+0000 7fabad16d640 10 monclient(hunting): picked mon.noname-c con 0x7faba80b1b00 addr v1:172.21.15.192:6789/0 2024-03-25T19:21:59.387 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.386+0000 7fabad16d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faba80a96f0 con 0x7faba80b44b0 2024-03-25T19:21:59.387 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.386+0000 7fabad16d640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faba80ab0e0 con 0x7faba80b1b00 2024-03-25T19:21:59.387 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.386+0000 7fabad16d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faba8145f70 con 0x7faba80b4e70 2024-03-25T19:21:59.387 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.386+0000 7fabad16d640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:59.387 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.386+0000 7fabad16d640 10 monclient(hunting): authenticate will time out at 6423.626466s 2024-03-25T19:21:59.398 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.398+0000 7faba6d9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7faba80b4e70 0x7faba813f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:38818/0 (socket says 172.21.15.6:38818) 2024-03-25T19:21:59.398 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.398+0000 7faba6d9d640 1 -- 172.21.15.6:0/1513842581 learned_addr learned my addr 172.21.15.6:0/1513842581 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:21:59.411 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.411+0000 7faba5d9b640 1 -- 172.21.15.6:0/1513842581 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1552935443 0 0) 0x7faba80ab0e0 con 0x7faba80b1b00 2024-03-25T19:21:59.411 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.411+0000 7faba5d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:21:59.411 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.411+0000 7faba5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:59.411 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.411+0000 7faba5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:59.411 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.411+0000 7faba5d9b640 10 monclient(hunting): my global_id is 4585 2024-03-25T19:21:59.411 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.411+0000 7faba5d9b640 1 -- 172.21.15.6:0/1513842581 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab90003520 con 0x7faba80b1b00 2024-03-25T19:21:59.422 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.422+0000 7faba5d9b640 1 -- 172.21.15.6:0/1513842581 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4185202638 0 0) 0x7faba8145f70 con 0x7faba80b4e70 2024-03-25T19:21:59.422 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.422+0000 7faba5d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:21:59.423 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.422+0000 7faba5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:59.423 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.422+0000 7faba5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:59.423 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.422+0000 7faba5d9b640 10 monclient(hunting): my global_id is 4580 2024-03-25T19:21:59.423 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.422+0000 7faba5d9b640 1 -- 172.21.15.6:0/1513842581 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faba80ab0e0 con 0x7faba80b4e70 2024-03-25T19:21:59.425 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.425+0000 7faba5d9b640 1 -- 172.21.15.6:0/1513842581 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1997182769 0 0) 0x7faba80a96f0 con 0x7faba80b44b0 2024-03-25T19:21:59.425 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.425+0000 7faba5d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:21:59.425 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.425+0000 7faba5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:59.425 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.425+0000 7faba5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:59.425 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.425+0000 7faba5d9b640 10 monclient(hunting): my global_id is 4572 2024-03-25T19:21:59.425 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.425+0000 7faba5d9b640 1 -- 172.21.15.6:0/1513842581 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faba8145f70 con 0x7faba80b44b0 2024-03-25T19:21:59.436 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.436+0000 7faba5d9b640 1 -- 172.21.15.6:0/1513842581 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2068948337 0 0) 0x7fab90003520 con 0x7faba80b1b00 2024-03-25T19:21:59.436 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.436+0000 7faba5d9b640 1 -- 172.21.15.6:0/1513842581 >> v1:172.21.15.6:6790/0 conn(0x7faba80b4e70 legacy=0x7faba813f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:59.436 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.436+0000 7faba5d9b640 1 -- 172.21.15.6:0/1513842581 >> v1:172.21.15.6:6789/0 conn(0x7faba80b44b0 legacy=0x7faba8145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:59.436 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.436+0000 7faba5d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:21:59.436 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.436+0000 7faba5d9b640 1 monclient: found mon.noname-c 2024-03-25T19:21:59.436 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.436+0000 7faba5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:59.436 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.436+0000 7faba5d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:21:59.436 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.436+0000 7faba5d9b640 1 -- 172.21.15.6:0/1513842581 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba80a9b80 con 0x7faba80b1b00 2024-03-25T19:21:59.436 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.436+0000 7faba5d9b640 10 monclient: _finish_auth 0 2024-03-25T19:21:59.437 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.436+0000 7faba5d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:21:59.437 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.436+0000 7faba5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:59.437 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.436+0000 7faba5d9b640 1 -- 172.21.15.6:0/1513842581 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7faba0001a10 con 0x7faba80b1b00 2024-03-25T19:21:59.437 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.436+0000 7fabad16d640 5 monclient: authenticate success, global_id 4585 2024-03-25T19:21:59.437 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.436+0000 7fabad16d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:21:59.437 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.436+0000 7faba5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:59.437 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.436+0000 7faba5d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:21:59.437 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.436+0000 7faba5d9b640 10 monclient: dump: 2024-03-25T19:21:59.437 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:21:59.437 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:59.437 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:59.437 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:59.437 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:59.437 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:59.437 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:21:59.437 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:21:59.438 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:21:59.438 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:59.438 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.436+0000 7fabad16d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:21:59.442 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.442+0000 7faba5d9b640 1 -- 172.21.15.6:0/1513842581 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faba0001b70 con 0x7faba80b1b00 2024-03-25T19:21:59.442 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.442+0000 7faba5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:59.442 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.442+0000 7faba5d9b640 1 -- 172.21.15.6:0/1513842581 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7faba0001f80 con 0x7faba80b1b00 2024-03-25T19:21:59.442 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.442+0000 7faba5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:59.442 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.442+0000 7faba5d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:21:59.442 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.442+0000 7faba5d9b640 10 monclient: dump: 2024-03-25T19:21:59.442 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:21:59.442 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:59.442 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:59.442 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:59.443 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:59.443 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:59.443 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:21:59.443 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:21:59.443 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:21:59.443 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:59.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.442+0000 7fabad16d640 10 monclient: get_monmap_and_config success 2024-03-25T19:21:59.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.442+0000 7fabad16d640 4 set_mon_vals no callback set 2024-03-25T19:21:59.445 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.443+0000 7fabad16d640 10 monclient: shutdown 2024-03-25T19:21:59.445 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.443+0000 7fabad16d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:59.445 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.443+0000 7fabad16d640 1 -- 172.21.15.6:0/1513842581 >> v1:172.21.15.192:6789/0 conn(0x7faba80b1b00 legacy=0x7faba814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:59.445 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.443+0000 7fabad16d640 1 -- 172.21.15.6:0/1513842581 shutdown_connections 2024-03-25T19:21:59.445 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.443+0000 7fabad16d640 1 -- 172.21.15.6:0/1513842581 >> 172.21.15.6:0/1513842581 conn(0x7faba80617a0 msgr2=0x7faba8061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:59.445 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.443+0000 7fabad16d640 1 -- 172.21.15.6:0/1513842581 shutdown_connections 2024-03-25T19:21:59.445 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.443+0000 7fabad16d640 1 -- 172.21.15.6:0/1513842581 wait complete. 2024-03-25T19:21:59.445 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.444+0000 7fabad16d640 10 monclient: build_initial_monmap 2024-03-25T19:21:59.445 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.444+0000 7fabad16d640 10 monclient: monmap: 2024-03-25T19:21:59.445 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:21:59.445 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:21:59.445 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:21:59.446 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:21:59.446 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:59.446 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:21:59.446 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:21:59.446 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:21:59.446 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:21:59.446 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:59.446 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.444+0000 7fabad16d640 1 librados: starting msgr at 2024-03-25T19:21:59.446 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.444+0000 7fabad16d640 1 librados: starting objecter 2024-03-25T19:21:59.446 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.444+0000 7fabad16d640 1 Processor -- start 2024-03-25T19:21:59.446 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.444+0000 7fabad16d640 1 -- start start 2024-03-25T19:21:59.446 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.444+0000 7fabad16d640 1 librados: setting wanted keys 2024-03-25T19:21:59.446 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.444+0000 7fabad16d640 1 librados: calling monclient init 2024-03-25T19:21:59.446 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.444+0000 7fabad16d640 10 monclient: init 2024-03-25T19:21:59.447 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.445+0000 7fabad16d640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:59.447 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.445+0000 7fabad16d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:21:59.447 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.445+0000 7fabad16d640 10 monclient(hunting): picked mon.noname-a con 0x7faba80b1b00 addr v1:172.21.15.6:6789/0 2024-03-25T19:21:59.447 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.445+0000 7fabad16d640 10 monclient(hunting): picked mon.noname-c con 0x7faba80b44b0 addr v1:172.21.15.6:6790/0 2024-03-25T19:21:59.447 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.445+0000 7fabad16d640 10 monclient(hunting): picked mon.noname-b con 0x7faba80b4e70 addr v1:172.21.15.192:6789/0 2024-03-25T19:21:59.447 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.445+0000 7fabad16d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faba8135a60 con 0x7faba80b1b00 2024-03-25T19:21:59.447 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.445+0000 7fabad16d640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faba8136e30 con 0x7faba80b4e70 2024-03-25T19:21:59.447 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.445+0000 7fabad16d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faba8135270 con 0x7faba80b44b0 2024-03-25T19:21:59.447 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.445+0000 7fabad16d640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:59.447 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.445+0000 7fabad16d640 10 monclient(hunting): authenticate will time out at 6423.685104s 2024-03-25T19:21:59.452 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.452+0000 7faba659c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7faba80b44b0 0x7faba81dac50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:38860/0 (socket says 172.21.15.6:38860) 2024-03-25T19:21:59.452 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.452+0000 7faba659c640 1 -- 172.21.15.6:0/839750025 learned_addr learned my addr 172.21.15.6:0/839750025 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:21:59.452 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.452+0000 7fab8f7fe640 10 client.?.objecter ms_handle_connect 0x7faba80b44b0 2024-03-25T19:21:59.452 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.452+0000 7fab8f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:59.461 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.461+0000 7fab8f7fe640 1 -- 172.21.15.6:0/839750025 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1885387647 0 0) 0x7faba8135270 con 0x7faba80b44b0 2024-03-25T19:21:59.461 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.461+0000 7fab8f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:21:59.461 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.461+0000 7fab8f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:59.461 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.461+0000 7fab8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:59.462 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.461+0000 7fab8f7fe640 10 monclient(hunting): my global_id is 4583 2024-03-25T19:21:59.462 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.461+0000 7fab8f7fe640 1 -- 172.21.15.6:0/839750025 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab94004eb0 con 0x7faba80b44b0 2024-03-25T19:21:59.469 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.469+0000 7fab8f7fe640 10 client.?.objecter ms_handle_connect 0x7faba80b4e70 2024-03-25T19:21:59.469 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.469+0000 7fab8f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:59.477 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.477+0000 7fab8f7fe640 10 client.?.objecter ms_handle_connect 0x7faba80b1b00 2024-03-25T19:21:59.477 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.477+0000 7fab8f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:59.480 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.480+0000 7fab8f7fe640 1 -- 172.21.15.6:0/839750025 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1888445052 0 0) 0x7fab94004eb0 con 0x7faba80b44b0 2024-03-25T19:21:59.480 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.480+0000 7fab8f7fe640 1 -- 172.21.15.6:0/839750025 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faba8135270 con 0x7faba80b44b0 2024-03-25T19:21:59.480 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.480+0000 7fab8f7fe640 1 -- 172.21.15.6:0/839750025 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fab9c0019d0 con 0x7faba80b44b0 2024-03-25T19:21:59.480 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.480+0000 7fab8f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:59.480 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.480+0000 7fab8f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:21:59.480 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.480+0000 7fab8f7fe640 10 monclient(hunting): dump: 2024-03-25T19:21:59.480 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:21:59.481 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:59.481 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:59.481 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:59.481 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:59.481 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:59.481 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:21:59.481 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:21:59.481 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:21:59.481 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:59.481 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.480+0000 7fab8f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:21:59.481 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.480+0000 7fab8f7fe640 1 -- 172.21.15.6:0/839750025 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 627481207 0 0) 0x7faba8136e30 con 0x7faba80b4e70 2024-03-25T19:21:59.481 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.480+0000 7fab8f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:21:59.481 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.480+0000 7fab8f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:59.481 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.480+0000 7fab8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:59.481 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.480+0000 7fab8f7fe640 10 monclient(hunting): my global_id is 4588 2024-03-25T19:21:59.481 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.480+0000 7fab8f7fe640 1 -- 172.21.15.6:0/839750025 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab94004eb0 con 0x7faba80b4e70 2024-03-25T19:21:59.492 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.492+0000 7fab8f7fe640 1 -- 172.21.15.6:0/839750025 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1785649760 0 0) 0x7faba8135a60 con 0x7faba80b1b00 2024-03-25T19:21:59.492 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.492+0000 7fab8f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:21:59.493 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.492+0000 7fab8f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:59.493 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.492+0000 7fab8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:59.493 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.492+0000 7fab8f7fe640 10 monclient(hunting): my global_id is 4575 2024-03-25T19:21:59.493 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.492+0000 7fab8f7fe640 1 -- 172.21.15.6:0/839750025 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faba8136e30 con 0x7faba80b1b00 2024-03-25T19:21:59.495 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.495+0000 7fab8f7fe640 1 -- 172.21.15.6:0/839750025 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2987461180 0 0) 0x7faba8135270 con 0x7faba80b44b0 2024-03-25T19:21:59.495 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.495+0000 7fab8f7fe640 1 -- 172.21.15.6:0/839750025 >> v1:172.21.15.192:6789/0 conn(0x7faba80b4e70 legacy=0x7faba81db360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:59.495 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.495+0000 7fab8f7fe640 1 -- 172.21.15.6:0/839750025 >> v1:172.21.15.6:6789/0 conn(0x7faba80b1b00 legacy=0x7faba8134a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:59.496 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.495+0000 7fab8f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:21:59.496 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.495+0000 7fab8f7fe640 1 monclient: found mon.c 2024-03-25T19:21:59.496 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.495+0000 7fab8f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:59.496 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.495+0000 7fab8f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:21:59.496 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.495+0000 7fab8f7fe640 1 -- 172.21.15.6:0/839750025 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba8135400 con 0x7faba80b44b0 2024-03-25T19:21:59.496 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.495+0000 7fab8f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:21:59.496 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.495+0000 7fab8f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:21:59.496 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.495+0000 7fab8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:59.496 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.495+0000 7fabad16d640 5 monclient: authenticate success, global_id 4583 2024-03-25T19:21:59.496 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.495+0000 7fabad16d640 10 monclient: _renew_subs 2024-03-25T19:21:59.496 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.495+0000 7fabad16d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:21:59.496 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.495+0000 7fabad16d640 1 -- 172.21.15.6:0/839750025 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faba8135650 con 0x7faba80b44b0 2024-03-25T19:21:59.496 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.495+0000 7fabad16d640 10 client.4583.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:21:59.496 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.495+0000 7fabad16d640 10 monclient: _renew_subs 2024-03-25T19:21:59.496 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.495+0000 7fabad16d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:21:59.497 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.495+0000 7fabad16d640 1 -- 172.21.15.6:0/839750025 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faba81e3380 con 0x7faba80b44b0 2024-03-25T19:21:59.497 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.495+0000 7fabad16d640 1 librados: init done 2024-03-25T19:21:59.497 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.496+0000 7fabad16d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:59.497 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.496+0000 7fabad16d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:59.497 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.496+0000 7fabad16d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:21:59.497 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.496+0000 7fabad16d640 1 -- 172.21.15.6:0/839750025 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faba804f730 con 0x7faba80b44b0 2024-03-25T19:21:59.832 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.831+0000 7fab8f7fe640 1 -- 172.21.15.6:0/839750025 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab9c001c50 con 0x7faba80b44b0 2024-03-25T19:21:59.832 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.831+0000 7fab8f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:59.832 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.831+0000 7fab8f7fe640 1 -- 172.21.15.6:0/839750025 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fab9c003a60 con 0x7faba80b44b0 2024-03-25T19:21:59.832 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.831+0000 7fab8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:59.832 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.831+0000 7fab8f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:21:59.832 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.831+0000 7fab8f7fe640 10 monclient: dump: 2024-03-25T19:21:59.832 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:21:59.832 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:21:59.832 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:59.832 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:21:59.832 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:59.832 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:21:59.832 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:21:59.832 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:21:59.833 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:21:59.833 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:21:59.833 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.831+0000 7faba559a640 4 set_mon_vals no callback set 2024-03-25T19:21:59.833 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.832+0000 7fab8f7fe640 1 -- 172.21.15.6:0/839750025 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7fab9c003c80 con 0x7faba80b44b0 2024-03-25T19:21:59.833 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.833+0000 7faba6d9d640 10 monclient: get_auth_request con 0x7fab9406de10 auth_method 0 2024-03-25T19:21:59.833 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.833+0000 7fab8f7fe640 10 client.4583.objecter ms_handle_connect 0x7fab9406de10 2024-03-25T19:21:59.838 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.838+0000 7fab8f7fe640 1 -- 172.21.15.6:0/839750025 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7fab9c0548b0 con 0x7faba80b44b0 2024-03-25T19:21:59.838 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.838+0000 7fab8f7fe640 10 client.4583.objecter ms_dispatch 0x7faba8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:21:59.838 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.838+0000 7fab8f7fe640 3 client.4583.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:21:59.838 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.838+0000 7fab8f7fe640 3 client.4583.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:21:59.838 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.838+0000 7fab8f7fe640 20 client.4583.objecter dump_active .. 0 homeless 2024-03-25T19:21:59.838 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.838+0000 7fab8f7fe640 1 -- 172.21.15.6:0/839750025 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fab9c0501f0 con 0x7faba80b44b0 2024-03-25T19:21:59.838 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.838+0000 7fab8f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:59.838 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:21:59.838+0000 7fab8f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:22:00.073 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:00.073+0000 7fabad16d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:00.073 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:00.073+0000 7fabad16d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:00.073 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:00.073+0000 7fabad16d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:22:00.073 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:00.073+0000 7fabad16d640 1 -- 172.21.15.6:0/839750025 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faba80aa090 con 0x7faba80b44b0 2024-03-25T19:22:00.114 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:00.114+0000 7fab8f7fe640 1 -- 172.21.15.6:0/839750025 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7fab9c0522f0 con 0x7faba80b44b0 2024-03-25T19:22:00.114 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:00.114+0000 7fab8f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:00.114 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:00.114+0000 7fab8f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:22:00.115 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:22:00.115 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:22:00.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:00.117+0000 7fabad16d640 10 librados: watch_flush enter 2024-03-25T19:22:00.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:00.117+0000 7fabad16d640 10 librados: watch_flush exit 2024-03-25T19:22:00.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:00.117+0000 7fabad16d640 20 client.4583.objecter shutdown clearing up homeless session... 2024-03-25T19:22:00.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:00.117+0000 7fabad16d640 10 client.4583.objecter successfully canceled tick 2024-03-25T19:22:00.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:00.117+0000 7fabad16d640 1 -- 172.21.15.6:0/839750025 >> v1:172.21.15.6:6800/2936652650 conn(0x7fab9406de10 legacy=0x7fab940702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:00.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:00.117+0000 7fabad16d640 10 monclient: shutdown 2024-03-25T19:22:00.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:00.117+0000 7fabad16d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:00.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:00.117+0000 7fabad16d640 1 -- 172.21.15.6:0/839750025 >> v1:172.21.15.6:6790/0 conn(0x7faba80b44b0 legacy=0x7faba81dac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:00.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:00.117+0000 7fabad16d640 1 -- 172.21.15.6:0/839750025 shutdown_connections 2024-03-25T19:22:00.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:00.117+0000 7fabad16d640 1 -- 172.21.15.6:0/839750025 >> 172.21.15.6:0/839750025 conn(0x7faba80617a0 msgr2=0x7faba80b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:00.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:00.117+0000 7fabad16d640 1 -- 172.21.15.6:0/839750025 shutdown_connections 2024-03-25T19:22:00.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:00.117+0000 7fabad16d640 1 -- 172.21.15.6:0/839750025 wait complete. 2024-03-25T19:22:00.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:00.117+0000 7fabad16d640 1 librados: shutdown 2024-03-25T19:22:00.130 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:22:06.937 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:22:07.049 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.049+0000 7fedeb655640 10 monclient: get_monmap_and_config 2024-03-25T19:22:07.049 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.049+0000 7fedeb655640 10 monclient: build_initial_monmap 2024-03-25T19:22:07.049 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.049+0000 7fedeb655640 10 monclient: monmap: 2024-03-25T19:22:07.049 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:22:07.049 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:07.049 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:22:07.050168+0000 2024-03-25T19:22:07.049 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:22:07.050168+0000 2024-03-25T19:22:07.050 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:07.050 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:07.050 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:22:07.050 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:22:07.050 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:22:07.050 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:07.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.049+0000 7fedeb655640 1 Processor -- start 2024-03-25T19:22:07.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.050+0000 7fedeb655640 1 -- start start 2024-03-25T19:22:07.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.050+0000 7fedeb655640 10 monclient: init 2024-03-25T19:22:07.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.050+0000 7fedeb655640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:07.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.050+0000 7fedeb655640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:22:07.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.050+0000 7fedeb655640 10 monclient(hunting): picked mon.noname-c con 0x7fede40b4ff0 addr v1:172.21.15.192:6789/0 2024-03-25T19:22:07.052 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.050+0000 7fedeb655640 10 monclient(hunting): picked mon.noname-a con 0x7fede40b4670 addr v1:172.21.15.6:6789/0 2024-03-25T19:22:07.052 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.051+0000 7fedeb655640 10 monclient(hunting): picked mon.noname-b con 0x7fede413c480 addr v1:172.21.15.6:6790/0 2024-03-25T19:22:07.052 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.051+0000 7fedeb655640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fede40ab0e0 con 0x7fede40b4670 2024-03-25T19:22:07.052 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.051+0000 7fedeb655640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fede40ab3c0 con 0x7fede40b4ff0 2024-03-25T19:22:07.052 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.051+0000 7fedeb655640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fede40a9f30 con 0x7fede413c480 2024-03-25T19:22:07.052 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.051+0000 7fedeb655640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:07.052 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.051+0000 7fedeb655640 10 monclient(hunting): authenticate will time out at 6431.290926s 2024-03-25T19:22:07.054 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.055+0000 7fede93f1640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fede40b4ff0 0x7fede413bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:41508/0 (socket says 172.21.15.6:41508) 2024-03-25T19:22:07.055 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.055+0000 7fede93f1640 1 -- 172.21.15.6:0/3056689334 learned_addr learned my addr 172.21.15.6:0/3056689334 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:22:07.070 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.070+0000 7feddbfff640 1 -- 172.21.15.6:0/3056689334 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2103813596 0 0) 0x7fede40ab3c0 con 0x7fede40b4ff0 2024-03-25T19:22:07.071 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.070+0000 7feddbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:22:07.071 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.070+0000 7feddbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:07.071 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.070+0000 7feddbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:07.071 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.070+0000 7feddbfff640 10 monclient(hunting): my global_id is 4591 2024-03-25T19:22:07.071 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.071+0000 7feddbfff640 1 -- 172.21.15.6:0/3056689334 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedd0003520 con 0x7fede40b4ff0 2024-03-25T19:22:07.081 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.080+0000 7feddbfff640 1 -- 172.21.15.6:0/3056689334 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2375856212 0 0) 0x7fedd0003520 con 0x7fede40b4ff0 2024-03-25T19:22:07.081 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.080+0000 7feddbfff640 1 -- 172.21.15.6:0/3056689334 >> v1:172.21.15.6:6790/0 conn(0x7fede413c480 legacy=0x7fede414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:07.081 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.080+0000 7feddbfff640 1 -- 172.21.15.6:0/3056689334 >> v1:172.21.15.6:6789/0 conn(0x7fede40b4670 legacy=0x7fede41425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:07.081 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.080+0000 7feddbfff640 10 monclient: _finish_hunting 0 2024-03-25T19:22:07.081 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.080+0000 7feddbfff640 1 monclient: found mon.noname-c 2024-03-25T19:22:07.081 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.080+0000 7feddbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:07.081 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.080+0000 7feddbfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:22:07.081 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.080+0000 7feddbfff640 1 -- 172.21.15.6:0/3056689334 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede40a9b70 con 0x7fede40b4ff0 2024-03-25T19:22:07.081 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.080+0000 7feddbfff640 10 monclient: _finish_auth 0 2024-03-25T19:22:07.081 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.080+0000 7feddbfff640 10 monclient: _check_auth_tickets 2024-03-25T19:22:07.082 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.080+0000 7feddbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:07.082 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.080+0000 7feddbfff640 1 -- 172.21.15.6:0/3056689334 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fede0001690 con 0x7fede40b4ff0 2024-03-25T19:22:07.082 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.080+0000 7feddbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:07.082 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.080+0000 7feddbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:22:07.082 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.080+0000 7feddbfff640 10 monclient: dump: 2024-03-25T19:22:07.082 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:22:07.082 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:07.082 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:07.082 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:07.082 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:07.082 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:07.082 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:22:07.082 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:22:07.082 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:22:07.082 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:07.083 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.081+0000 7fedeb655640 5 monclient: authenticate success, global_id 4591 2024-03-25T19:22:07.083 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.081+0000 7fedeb655640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:07.097 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.097+0000 7feddbfff640 1 -- 172.21.15.6:0/3056689334 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fede00017f0 con 0x7fede40b4ff0 2024-03-25T19:22:07.097 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.097+0000 7feddbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:07.097 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.097+0000 7feddbfff640 1 -- 172.21.15.6:0/3056689334 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fede0001c00 con 0x7fede40b4ff0 2024-03-25T19:22:07.097 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.097+0000 7fedeb655640 10 monclient: get_monmap_and_config success 2024-03-25T19:22:07.097 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.097+0000 7fedeb655640 4 set_mon_vals no callback set 2024-03-25T19:22:07.097 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.097+0000 7feddbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:07.097 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.097+0000 7feddbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:07.098 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.097+0000 7feddbfff640 10 monclient: dump: 2024-03-25T19:22:07.098 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:22:07.098 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:07.098 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:07.098 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:07.098 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:07.098 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:07.098 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:22:07.098 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:22:07.098 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:22:07.098 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:07.100 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.098+0000 7fedeb655640 10 monclient: shutdown 2024-03-25T19:22:07.100 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.098+0000 7fedeb655640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:07.100 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.098+0000 7fedeb655640 1 -- 172.21.15.6:0/3056689334 >> v1:172.21.15.192:6789/0 conn(0x7fede40b4ff0 legacy=0x7fede413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:07.101 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.098+0000 7fedeb655640 1 -- 172.21.15.6:0/3056689334 shutdown_connections 2024-03-25T19:22:07.101 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.098+0000 7fedeb655640 1 -- 172.21.15.6:0/3056689334 >> 172.21.15.6:0/3056689334 conn(0x7fede40617a0 msgr2=0x7fede4061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:07.101 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.098+0000 7fedeb655640 1 -- 172.21.15.6:0/3056689334 shutdown_connections 2024-03-25T19:22:07.101 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.098+0000 7fedeb655640 1 -- 172.21.15.6:0/3056689334 wait complete. 2024-03-25T19:22:07.101 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.099+0000 7fedeb655640 10 monclient: build_initial_monmap 2024-03-25T19:22:07.101 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.099+0000 7fedeb655640 10 monclient: monmap: 2024-03-25T19:22:07.101 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:22:07.101 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:22:07.101 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:22:07.101 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:22:07.101 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:07.101 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:22:07.101 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:22:07.101 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:22:07.102 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:22:07.102 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:07.102 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.099+0000 7fedeb655640 1 librados: starting msgr at 2024-03-25T19:22:07.102 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.099+0000 7fedeb655640 1 librados: starting objecter 2024-03-25T19:22:07.102 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.099+0000 7fedeb655640 1 Processor -- start 2024-03-25T19:22:07.102 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.099+0000 7fedeb655640 1 -- start start 2024-03-25T19:22:07.102 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.099+0000 7fedeb655640 1 librados: setting wanted keys 2024-03-25T19:22:07.102 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.099+0000 7fedeb655640 1 librados: calling monclient init 2024-03-25T19:22:07.102 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.099+0000 7fedeb655640 10 monclient: init 2024-03-25T19:22:07.102 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.099+0000 7fedeb655640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:07.102 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.099+0000 7fedeb655640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:22:07.102 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.100+0000 7fedeb655640 10 monclient(hunting): picked mon.noname-c con 0x7fede40b4670 addr v1:172.21.15.6:6790/0 2024-03-25T19:22:07.102 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.100+0000 7fedeb655640 10 monclient(hunting): picked mon.noname-a con 0x7fede40b4ff0 addr v1:172.21.15.6:6789/0 2024-03-25T19:22:07.102 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.100+0000 7fedeb655640 10 monclient(hunting): picked mon.noname-b con 0x7fede413c480 addr v1:172.21.15.192:6789/0 2024-03-25T19:22:07.102 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.100+0000 7fedeb655640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fede40abc10 con 0x7fede40b4ff0 2024-03-25T19:22:07.103 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.100+0000 7fedeb655640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fede40ac380 con 0x7fede413c480 2024-03-25T19:22:07.103 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.100+0000 7fedeb655640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fede40ac660 con 0x7fede40b4670 2024-03-25T19:22:07.103 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.100+0000 7fedeb655640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:07.103 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.100+0000 7fedeb655640 10 monclient(hunting): authenticate will time out at 6431.339926s 2024-03-25T19:22:07.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.116+0000 7fede9bf2640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fede413c480 0x7fede41e3a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:41544/0 (socket says 172.21.15.6:41544) 2024-03-25T19:22:07.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.116+0000 7fede9bf2640 1 -- 172.21.15.6:0/3529780431 learned_addr learned my addr 172.21.15.6:0/3529780431 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:22:07.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.119+0000 7fedd9ffb640 10 client.?.objecter ms_handle_connect 0x7fede40b4670 2024-03-25T19:22:07.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.119+0000 7fedd9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:07.120 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.120+0000 7fedd9ffb640 10 client.?.objecter ms_handle_connect 0x7fede413c480 2024-03-25T19:22:07.120 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.120+0000 7fedd9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:07.121 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.121+0000 7fedd9ffb640 10 client.?.objecter ms_handle_connect 0x7fede40b4ff0 2024-03-25T19:22:07.121 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.121+0000 7fedd9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:07.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.129+0000 7fedd9ffb640 1 -- 172.21.15.6:0/3529780431 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3027123657 0 0) 0x7fede40ac660 con 0x7fede40b4670 2024-03-25T19:22:07.130 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.129+0000 7fedd9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:22:07.130 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.129+0000 7fedd9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:07.130 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.129+0000 7fedd9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:07.130 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.129+0000 7fedd9ffb640 10 monclient(hunting): my global_id is 4589 2024-03-25T19:22:07.130 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.129+0000 7fedd9ffb640 1 -- 172.21.15.6:0/3529780431 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedc8004eb0 con 0x7fede40b4670 2024-03-25T19:22:07.130 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.130+0000 7fedd9ffb640 1 -- 172.21.15.6:0/3529780431 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2189320532 0 0) 0x7fede40ac380 con 0x7fede413c480 2024-03-25T19:22:07.130 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.130+0000 7fedd9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:22:07.131 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.130+0000 7fedd9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:07.131 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.130+0000 7fedd9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:07.131 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.130+0000 7fedd9ffb640 10 monclient(hunting): my global_id is 4594 2024-03-25T19:22:07.131 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.130+0000 7fedd9ffb640 1 -- 172.21.15.6:0/3529780431 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fede40ac660 con 0x7fede413c480 2024-03-25T19:22:07.145 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.145+0000 7fedd9ffb640 1 -- 172.21.15.6:0/3529780431 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2786960017 0 0) 0x7fedc8004eb0 con 0x7fede40b4670 2024-03-25T19:22:07.145 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.145+0000 7fedd9ffb640 1 -- 172.21.15.6:0/3529780431 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fede40ac380 con 0x7fede40b4670 2024-03-25T19:22:07.145 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.145+0000 7fedd9ffb640 1 -- 172.21.15.6:0/3529780431 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fede0001bc0 con 0x7fede40b4670 2024-03-25T19:22:07.145 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.145+0000 7fedd9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:07.145 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.145+0000 7fedd9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:22:07.145 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.145+0000 7fedd9ffb640 10 monclient(hunting): dump: 2024-03-25T19:22:07.145 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:22:07.146 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:07.146 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:07.146 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:07.146 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:07.146 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:07.146 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:22:07.146 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:22:07.146 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:22:07.146 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:07.146 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.145+0000 7fedd9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:22:07.150 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.150+0000 7fedd9ffb640 1 -- 172.21.15.6:0/3529780431 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3147288381 0 0) 0x7fede40abc10 con 0x7fede40b4ff0 2024-03-25T19:22:07.150 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.150+0000 7fedd9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:22:07.150 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.150+0000 7fedd9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:07.150 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.150+0000 7fedd9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:07.150 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.150+0000 7fedd9ffb640 10 monclient(hunting): my global_id is 4581 2024-03-25T19:22:07.150 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.150+0000 7fedd9ffb640 1 -- 172.21.15.6:0/3529780431 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedc8004eb0 con 0x7fede40b4ff0 2024-03-25T19:22:07.157 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.157+0000 7fedd9ffb640 1 -- 172.21.15.6:0/3529780431 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 474597505 0 0) 0x7fede40ac660 con 0x7fede413c480 2024-03-25T19:22:07.157 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.157+0000 7fedd9ffb640 1 -- 172.21.15.6:0/3529780431 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fede40abc10 con 0x7fede413c480 2024-03-25T19:22:07.157 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.157+0000 7fedd9ffb640 1 -- 172.21.15.6:0/3529780431 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7feddc001c10 con 0x7fede413c480 2024-03-25T19:22:07.157 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.157+0000 7fedd9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:07.157 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.157+0000 7fedd9ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:07.157 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.157+0000 7fedd9ffb640 10 monclient(hunting): dump: 2024-03-25T19:22:07.157 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:22:07.157 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:07.158 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:07.158 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:07.158 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:07.158 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:07.158 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:22:07.158 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:22:07.158 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:22:07.158 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:07.162 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.161+0000 7fedd9ffb640 1 -- 172.21.15.6:0/3529780431 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4070581131 0 0) 0x7fede40ac380 con 0x7fede40b4670 2024-03-25T19:22:07.162 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.161+0000 7fedd9ffb640 1 -- 172.21.15.6:0/3529780431 >> v1:172.21.15.192:6789/0 conn(0x7fede413c480 legacy=0x7fede41e3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:07.162 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.161+0000 7fedd9ffb640 1 -- 172.21.15.6:0/3529780431 >> v1:172.21.15.6:6789/0 conn(0x7fede40b4ff0 legacy=0x7fede41d81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:07.162 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.161+0000 7fedd9ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:22:07.162 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.161+0000 7fedd9ffb640 1 monclient: found mon.c 2024-03-25T19:22:07.162 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.161+0000 7fedd9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:07.162 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.161+0000 7fedd9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:22:07.162 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.161+0000 7fedd9ffb640 1 -- 172.21.15.6:0/3529780431 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede40ac940 con 0x7fede40b4670 2024-03-25T19:22:07.162 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.161+0000 7fedd9ffb640 10 monclient: _finish_auth 0 2024-03-25T19:22:07.162 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.161+0000 7fedd9ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:22:07.162 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.161+0000 7fedd9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:07.163 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.161+0000 7fedeb655640 5 monclient: authenticate success, global_id 4589 2024-03-25T19:22:07.163 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.161+0000 7fedeb655640 10 monclient: _renew_subs 2024-03-25T19:22:07.163 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.161+0000 7fedeb655640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:22:07.163 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.161+0000 7fedeb655640 1 -- 172.21.15.6:0/3529780431 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fede40ab8e0 con 0x7fede40b4670 2024-03-25T19:22:07.163 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.162+0000 7fedeb655640 10 client.4589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:22:07.163 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.162+0000 7fedeb655640 10 monclient: _renew_subs 2024-03-25T19:22:07.163 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.162+0000 7fedeb655640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:22:07.163 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.162+0000 7fedeb655640 1 -- 172.21.15.6:0/3529780431 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fede40ac130 con 0x7fede40b4670 2024-03-25T19:22:07.163 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.162+0000 7fedeb655640 1 librados: init done 2024-03-25T19:22:07.164 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.163+0000 7fedeb655640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:07.164 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.163+0000 7fedeb655640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:07.164 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.163+0000 7fedeb655640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:22:07.164 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.163+0000 7fedeb655640 1 -- 172.21.15.6:0/3529780431 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fede40b3fd0 con 0x7fede40b4670 2024-03-25T19:22:07.383 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.383+0000 7fedd9ffb640 1 -- 172.21.15.6:0/3529780431 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fede0004a90 con 0x7fede40b4670 2024-03-25T19:22:07.383 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.383+0000 7fedd9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:07.383 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.383+0000 7feddb7fe640 4 set_mon_vals no callback set 2024-03-25T19:22:07.473 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.473+0000 7fedd9ffb640 1 -- 172.21.15.6:0/3529780431 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fede0004da0 con 0x7fede40b4670 2024-03-25T19:22:07.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.473+0000 7fedd9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:07.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.473+0000 7fedd9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:22:07.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.473+0000 7fedd9ffb640 10 monclient: dump: 2024-03-25T19:22:07.474 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:22:07.474 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:07.474 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:07.474 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:07.474 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:07.474 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:07.474 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:22:07.474 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:22:07.474 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:22:07.475 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:07.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.474+0000 7fedd9ffb640 1 -- 172.21.15.6:0/3529780431 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7fede001ad70 con 0x7fede40b4670 2024-03-25T19:22:07.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.474+0000 7fede8bf0640 10 monclient: get_auth_request con 0x7fedc806df70 auth_method 0 2024-03-25T19:22:07.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.475+0000 7fedd9ffb640 10 client.4589.objecter ms_handle_connect 0x7fedc806df70 2024-03-25T19:22:07.481 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.480+0000 7fedd9ffb640 1 -- 172.21.15.6:0/3529780431 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7fede0054970 con 0x7fede40b4670 2024-03-25T19:22:07.481 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.480+0000 7fedd9ffb640 10 client.4589.objecter ms_dispatch 0x7fede4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:22:07.481 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.480+0000 7fedd9ffb640 3 client.4589.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:22:07.481 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.480+0000 7fedd9ffb640 3 client.4589.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:22:07.481 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.481+0000 7fedd9ffb640 20 client.4589.objecter dump_active .. 0 homeless 2024-03-25T19:22:07.481 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.481+0000 7fedd9ffb640 1 -- 172.21.15.6:0/3529780431 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fede00502b0 con 0x7fede40b4670 2024-03-25T19:22:07.481 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.481+0000 7fedd9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:07.481 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.481+0000 7fedd9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:22:07.714 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.714+0000 7fedeb655640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:07.714 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.714+0000 7fedeb655640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:07.714 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.714+0000 7fedeb655640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:22:07.715 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.714+0000 7fedeb655640 1 -- 172.21.15.6:0/3529780431 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fede40aba70 con 0x7fede40b4670 2024-03-25T19:22:07.746 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.746+0000 7fedd9ffb640 1 -- 172.21.15.6:0/3529780431 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7fede40aba70 con 0x7fede40b4670 2024-03-25T19:22:07.746 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.746+0000 7fedd9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:07.746 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.746+0000 7fedd9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:22:07.746 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:22:07.746 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:22:07.749 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.748+0000 7fedeb655640 10 librados: watch_flush enter 2024-03-25T19:22:07.749 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.748+0000 7fedeb655640 10 librados: watch_flush exit 2024-03-25T19:22:07.749 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.748+0000 7fedeb655640 20 client.4589.objecter shutdown clearing up homeless session... 2024-03-25T19:22:07.749 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.748+0000 7fedeb655640 10 client.4589.objecter successfully canceled tick 2024-03-25T19:22:07.749 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.748+0000 7fedeb655640 1 -- 172.21.15.6:0/3529780431 >> v1:172.21.15.6:6800/2936652650 conn(0x7fedc806df70 legacy=0x7fedc8070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:07.749 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.748+0000 7fedeb655640 10 monclient: shutdown 2024-03-25T19:22:07.749 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.748+0000 7fedeb655640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:07.749 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.748+0000 7fedeb655640 1 -- 172.21.15.6:0/3529780431 >> v1:172.21.15.6:6790/0 conn(0x7fede40b4670 legacy=0x7fede4137070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:07.749 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.748+0000 7fedeb655640 1 -- 172.21.15.6:0/3529780431 shutdown_connections 2024-03-25T19:22:07.749 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.748+0000 7fedeb655640 1 -- 172.21.15.6:0/3529780431 >> 172.21.15.6:0/3529780431 conn(0x7fede40617a0 msgr2=0x7fede4145420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:07.750 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.748+0000 7fedeb655640 1 -- 172.21.15.6:0/3529780431 shutdown_connections 2024-03-25T19:22:07.750 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.748+0000 7fedeb655640 1 -- 172.21.15.6:0/3529780431 wait complete. 2024-03-25T19:22:07.750 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:07.748+0000 7fedeb655640 1 librados: shutdown 2024-03-25T19:22:07.760 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:22:09.168 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:08:26.514 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:09.184 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:08:26.531 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:09.217 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:08:26.564 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:09.228 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:08:26.575 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:09.237 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:08:26.583 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:09.237 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:08:26.555 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:09.242 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:08:26.560 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:09.268 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:08:26.614 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:09.271 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:08:26.589 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:14.567 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:22:14.685 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.685+0000 7fed4c316640 10 monclient: get_monmap_and_config 2024-03-25T19:22:14.686 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.685+0000 7fed4c316640 10 monclient: build_initial_monmap 2024-03-25T19:22:14.686 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.685+0000 7fed4c316640 10 monclient: monmap: 2024-03-25T19:22:14.686 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:22:14.686 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:14.686 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:22:14.686926+0000 2024-03-25T19:22:14.686 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:22:14.686926+0000 2024-03-25T19:22:14.686 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:14.686 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:14.686 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:22:14.686 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:22:14.686 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:22:14.686 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:14.688 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.686+0000 7fed4c316640 1 Processor -- start 2024-03-25T19:22:14.688 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.686+0000 7fed4c316640 1 -- start start 2024-03-25T19:22:14.688 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.686+0000 7fed4c316640 10 monclient: init 2024-03-25T19:22:14.688 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.687+0000 7fed4c316640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:14.688 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.687+0000 7fed4c316640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:22:14.688 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.687+0000 7fed4c316640 10 monclient(hunting): picked mon.noname-b con 0x7fed440b4e70 addr v1:172.21.15.6:6790/0 2024-03-25T19:22:14.688 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.687+0000 7fed4c316640 10 monclient(hunting): picked mon.noname-a con 0x7fed440b44b0 addr v1:172.21.15.6:6789/0 2024-03-25T19:22:14.688 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.687+0000 7fed4c316640 10 monclient(hunting): picked mon.noname-c con 0x7fed440b1b00 addr v1:172.21.15.192:6789/0 2024-03-25T19:22:14.688 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.687+0000 7fed4c316640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed440a96f0 con 0x7fed440b44b0 2024-03-25T19:22:14.688 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.687+0000 7fed4c316640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed440ab0e0 con 0x7fed440b1b00 2024-03-25T19:22:14.688 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.687+0000 7fed4c316640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed44145f70 con 0x7fed440b4e70 2024-03-25T19:22:14.688 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.687+0000 7fed4c316640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:14.689 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.687+0000 7fed4c316640 10 monclient(hunting): authenticate will time out at 6438.927225s 2024-03-25T19:22:14.703 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.703+0000 7fed4a8b3640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fed440b1b00 0x7fed4414dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:41454/0 (socket says 172.21.15.6:41454) 2024-03-25T19:22:14.703 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.703+0000 7fed4a8b3640 1 -- 172.21.15.6:0/3273369873 learned_addr learned my addr 172.21.15.6:0/3273369873 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:22:14.710 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.710+0000 7fed490b0640 1 -- 172.21.15.6:0/3273369873 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1303200861 0 0) 0x7fed440a96f0 con 0x7fed440b44b0 2024-03-25T19:22:14.710 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.710+0000 7fed490b0640 10 monclient(hunting): _negotiate 2024-03-25T19:22:14.710 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.710+0000 7fed490b0640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:14.711 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.710+0000 7fed490b0640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:14.711 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.710+0000 7fed490b0640 10 monclient(hunting): my global_id is 4584 2024-03-25T19:22:14.711 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.710+0000 7fed490b0640 1 -- 172.21.15.6:0/3273369873 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed30003520 con 0x7fed440b44b0 2024-03-25T19:22:14.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.723+0000 7fed490b0640 1 -- 172.21.15.6:0/3273369873 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 495946886 0 0) 0x7fed44145f70 con 0x7fed440b4e70 2024-03-25T19:22:14.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.723+0000 7fed490b0640 10 monclient(hunting): _negotiate 2024-03-25T19:22:14.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.723+0000 7fed490b0640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:14.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.723+0000 7fed490b0640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:14.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.723+0000 7fed490b0640 10 monclient(hunting): my global_id is 4592 2024-03-25T19:22:14.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.723+0000 7fed490b0640 1 -- 172.21.15.6:0/3273369873 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed440a96f0 con 0x7fed440b4e70 2024-03-25T19:22:14.724 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.724+0000 7fed490b0640 1 -- 172.21.15.6:0/3273369873 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1006119613 0 0) 0x7fed440ab0e0 con 0x7fed440b1b00 2024-03-25T19:22:14.724 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.724+0000 7fed490b0640 10 monclient(hunting): _negotiate 2024-03-25T19:22:14.724 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.724+0000 7fed490b0640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:14.724 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.724+0000 7fed490b0640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:14.724 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.724+0000 7fed490b0640 10 monclient(hunting): my global_id is 4597 2024-03-25T19:22:14.725 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.724+0000 7fed490b0640 1 -- 172.21.15.6:0/3273369873 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed44145f70 con 0x7fed440b1b00 2024-03-25T19:22:14.738 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.738+0000 7fed490b0640 1 -- 172.21.15.6:0/3273369873 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1953966155 0 0) 0x7fed44145f70 con 0x7fed440b1b00 2024-03-25T19:22:14.738 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.738+0000 7fed490b0640 1 -- 172.21.15.6:0/3273369873 >> v1:172.21.15.6:6790/0 conn(0x7fed440b4e70 legacy=0x7fed4413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:14.738 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.738+0000 7fed490b0640 1 -- 172.21.15.6:0/3273369873 >> v1:172.21.15.6:6789/0 conn(0x7fed440b44b0 legacy=0x7fed44145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:14.738 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.738+0000 7fed490b0640 10 monclient: _finish_hunting 0 2024-03-25T19:22:14.739 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.738+0000 7fed490b0640 1 monclient: found mon.noname-c 2024-03-25T19:22:14.739 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.738+0000 7fed490b0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:14.739 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.738+0000 7fed490b0640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:22:14.739 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.738+0000 7fed490b0640 1 -- 172.21.15.6:0/3273369873 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed440a9b80 con 0x7fed440b1b00 2024-03-25T19:22:14.739 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.738+0000 7fed490b0640 10 monclient: _finish_auth 0 2024-03-25T19:22:14.739 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.738+0000 7fed490b0640 10 monclient: _check_auth_tickets 2024-03-25T19:22:14.739 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.738+0000 7fed490b0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:14.739 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.738+0000 7fed490b0640 1 -- 172.21.15.6:0/3273369873 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fed3c001930 con 0x7fed440b1b00 2024-03-25T19:22:14.739 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.738+0000 7fed490b0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:14.739 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.738+0000 7fed490b0640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:22:14.739 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.738+0000 7fed490b0640 10 monclient: dump: 2024-03-25T19:22:14.739 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:22:14.739 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:14.739 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:14.739 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:14.740 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:14.740 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:14.740 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:22:14.740 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:22:14.740 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:22:14.740 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:14.740 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.738+0000 7fed4c316640 5 monclient: authenticate success, global_id 4597 2024-03-25T19:22:14.740 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.738+0000 7fed4c316640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:14.756 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.757+0000 7fed490b0640 1 -- 172.21.15.6:0/3273369873 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed3c001a90 con 0x7fed440b1b00 2024-03-25T19:22:14.757 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.757+0000 7fed490b0640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:14.757 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.757+0000 7fed490b0640 1 -- 172.21.15.6:0/3273369873 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fed3c001ea0 con 0x7fed440b1b00 2024-03-25T19:22:14.757 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.757+0000 7fed4c316640 10 monclient: get_monmap_and_config success 2024-03-25T19:22:14.757 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.757+0000 7fed4c316640 4 set_mon_vals no callback set 2024-03-25T19:22:14.757 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.757+0000 7fed490b0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:14.757 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.757+0000 7fed490b0640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:14.757 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.757+0000 7fed490b0640 10 monclient: dump: 2024-03-25T19:22:14.757 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:22:14.757 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:14.757 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:14.757 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:14.757 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:14.758 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:14.758 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:22:14.758 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:22:14.758 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:22:14.758 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:14.761 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.757+0000 7fed4c316640 10 monclient: shutdown 2024-03-25T19:22:14.761 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.757+0000 7fed4c316640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:14.761 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.757+0000 7fed4c316640 1 -- 172.21.15.6:0/3273369873 >> v1:172.21.15.192:6789/0 conn(0x7fed440b1b00 legacy=0x7fed4414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:14.761 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.758+0000 7fed4c316640 1 -- 172.21.15.6:0/3273369873 shutdown_connections 2024-03-25T19:22:14.761 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.758+0000 7fed4c316640 1 -- 172.21.15.6:0/3273369873 >> 172.21.15.6:0/3273369873 conn(0x7fed440617a0 msgr2=0x7fed44061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:14.761 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.758+0000 7fed4c316640 1 -- 172.21.15.6:0/3273369873 shutdown_connections 2024-03-25T19:22:14.761 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.758+0000 7fed4c316640 1 -- 172.21.15.6:0/3273369873 wait complete. 2024-03-25T19:22:14.761 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.758+0000 7fed4c316640 10 monclient: build_initial_monmap 2024-03-25T19:22:14.761 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.758+0000 7fed4c316640 10 monclient: monmap: 2024-03-25T19:22:14.761 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:22:14.761 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:22:14.761 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:22:14.761 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:22:14.762 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:14.762 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:22:14.762 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:22:14.762 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:22:14.762 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:22:14.762 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:14.762 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.759+0000 7fed4c316640 1 librados: starting msgr at 2024-03-25T19:22:14.762 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.759+0000 7fed4c316640 1 librados: starting objecter 2024-03-25T19:22:14.762 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.759+0000 7fed4c316640 1 Processor -- start 2024-03-25T19:22:14.762 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.759+0000 7fed4c316640 1 -- start start 2024-03-25T19:22:14.762 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.759+0000 7fed4c316640 1 librados: setting wanted keys 2024-03-25T19:22:14.762 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.759+0000 7fed4c316640 1 librados: calling monclient init 2024-03-25T19:22:14.762 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.759+0000 7fed4c316640 10 monclient: init 2024-03-25T19:22:14.762 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.759+0000 7fed4c316640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:14.763 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.759+0000 7fed4c316640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:22:14.763 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.759+0000 7fed4c316640 10 monclient(hunting): picked mon.noname-a con 0x7fed440b1b00 addr v1:172.21.15.6:6789/0 2024-03-25T19:22:14.763 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.759+0000 7fed4c316640 10 monclient(hunting): picked mon.noname-c con 0x7fed440b44b0 addr v1:172.21.15.6:6790/0 2024-03-25T19:22:14.763 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.760+0000 7fed4c316640 10 monclient(hunting): picked mon.noname-b con 0x7fed440b4e70 addr v1:172.21.15.192:6789/0 2024-03-25T19:22:14.763 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.760+0000 7fed4c316640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed441d5550 con 0x7fed440b1b00 2024-03-25T19:22:14.763 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.760+0000 7fed4c316640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed441d7a70 con 0x7fed440b4e70 2024-03-25T19:22:14.763 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.760+0000 7fed4c316640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed441d86a0 con 0x7fed440b44b0 2024-03-25T19:22:14.763 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.760+0000 7fed4c316640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:14.763 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.760+0000 7fed4c316640 10 monclient(hunting): authenticate will time out at 6438.999911s 2024-03-25T19:22:14.773 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.773+0000 7fed4a8b3640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fed440b4e70 0x7fed441e2190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:41510/0 (socket says 172.21.15.6:41510) 2024-03-25T19:22:14.773 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.773+0000 7fed4a8b3640 1 -- 172.21.15.6:0/1875239346 learned_addr learned my addr 172.21.15.6:0/1875239346 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:22:14.774 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.774+0000 7fed3affd640 10 client.?.objecter ms_handle_connect 0x7fed440b4e70 2024-03-25T19:22:14.774 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.774+0000 7fed3affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:14.776 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.776+0000 7fed3affd640 10 client.?.objecter ms_handle_connect 0x7fed440b44b0 2024-03-25T19:22:14.776 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.776+0000 7fed3affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:14.781 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.781+0000 7fed3affd640 10 client.?.objecter ms_handle_connect 0x7fed440b1b00 2024-03-25T19:22:14.781 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.781+0000 7fed3affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:14.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.783+0000 7fed3affd640 1 -- 172.21.15.6:0/1875239346 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3404503560 0 0) 0x7fed441d86a0 con 0x7fed440b44b0 2024-03-25T19:22:14.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.783+0000 7fed3affd640 10 monclient(hunting): _negotiate 2024-03-25T19:22:14.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.783+0000 7fed3affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:14.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.783+0000 7fed3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:14.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.783+0000 7fed3affd640 10 monclient(hunting): my global_id is 4595 2024-03-25T19:22:14.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.783+0000 7fed3affd640 1 -- 172.21.15.6:0/1875239346 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed28004eb0 con 0x7fed440b44b0 2024-03-25T19:22:14.789 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.789+0000 7fed3affd640 1 -- 172.21.15.6:0/1875239346 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2189930469 0 0) 0x7fed441d5550 con 0x7fed440b1b00 2024-03-25T19:22:14.789 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.789+0000 7fed3affd640 10 monclient(hunting): _negotiate 2024-03-25T19:22:14.789 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.789+0000 7fed3affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:14.789 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.789+0000 7fed3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:14.789 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.789+0000 7fed3affd640 10 monclient(hunting): my global_id is 4587 2024-03-25T19:22:14.789 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.789+0000 7fed3affd640 1 -- 172.21.15.6:0/1875239346 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed441d86a0 con 0x7fed440b1b00 2024-03-25T19:22:14.789 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.789+0000 7fed3affd640 1 -- 172.21.15.6:0/1875239346 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3515896178 0 0) 0x7fed441d7a70 con 0x7fed440b4e70 2024-03-25T19:22:14.790 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.789+0000 7fed3affd640 10 monclient(hunting): _negotiate 2024-03-25T19:22:14.790 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.789+0000 7fed3affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:14.790 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.789+0000 7fed3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:14.790 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.789+0000 7fed3affd640 10 monclient(hunting): my global_id is 4600 2024-03-25T19:22:14.790 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.789+0000 7fed3affd640 1 -- 172.21.15.6:0/1875239346 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed441d5550 con 0x7fed440b4e70 2024-03-25T19:22:14.807 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.807+0000 7fed3affd640 1 -- 172.21.15.6:0/1875239346 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1222094514 0 0) 0x7fed441d5550 con 0x7fed440b4e70 2024-03-25T19:22:14.807 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.807+0000 7fed3affd640 1 -- 172.21.15.6:0/1875239346 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed441d7a70 con 0x7fed440b4e70 2024-03-25T19:22:14.807 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.807+0000 7fed3affd640 1 -- 172.21.15.6:0/1875239346 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fed3c001da0 con 0x7fed440b4e70 2024-03-25T19:22:14.807 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.807+0000 7fed3affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:14.808 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.807+0000 7fed3affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:22:14.808 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.807+0000 7fed3affd640 10 monclient(hunting): dump: 2024-03-25T19:22:14.808 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:22:14.808 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:14.808 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:14.808 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:14.808 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:14.808 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:14.808 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:22:14.808 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:22:14.808 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:22:14.808 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:14.808 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.807+0000 7fed3affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:22:14.814 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.814+0000 7fed3affd640 1 -- 172.21.15.6:0/1875239346 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 977477844 0 0) 0x7fed28004eb0 con 0x7fed440b44b0 2024-03-25T19:22:14.815 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.814+0000 7fed3affd640 1 -- 172.21.15.6:0/1875239346 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed441d5550 con 0x7fed440b44b0 2024-03-25T19:22:14.815 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.814+0000 7fed3affd640 1 -- 172.21.15.6:0/1875239346 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fed34001950 con 0x7fed440b44b0 2024-03-25T19:22:14.815 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.814+0000 7fed3affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:14.815 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.814+0000 7fed3affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:22:14.815 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.814+0000 7fed3affd640 10 monclient(hunting): dump: 2024-03-25T19:22:14.815 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:22:14.815 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:14.815 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:14.815 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:14.815 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:14.815 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:14.815 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:22:14.816 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:22:14.816 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:22:14.816 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:14.827 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.827+0000 7fed3affd640 1 -- 172.21.15.6:0/1875239346 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2091569680 0 0) 0x7fed441d86a0 con 0x7fed440b1b00 2024-03-25T19:22:14.827 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.827+0000 7fed3affd640 1 -- 172.21.15.6:0/1875239346 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed28004eb0 con 0x7fed440b1b00 2024-03-25T19:22:14.827 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.827+0000 7fed3affd640 1 -- 172.21.15.6:0/1875239346 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fed40001810 con 0x7fed440b1b00 2024-03-25T19:22:14.827 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.827+0000 7fed3affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:14.827 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.827+0000 7fed3affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:14.827 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.827+0000 7fed3affd640 10 monclient(hunting): dump: 2024-03-25T19:22:14.827 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:22:14.827 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:14.827 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:14.827 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:14.828 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:14.828 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:14.828 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:22:14.828 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:22:14.828 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:22:14.828 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:14.831 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.830+0000 7fed3affd640 1 -- 172.21.15.6:0/1875239346 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1747847608 0 0) 0x7fed441d7a70 con 0x7fed440b4e70 2024-03-25T19:22:14.831 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.830+0000 7fed3affd640 1 -- 172.21.15.6:0/1875239346 >> v1:172.21.15.6:6790/0 conn(0x7fed440b44b0 legacy=0x7fed441367c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:14.831 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.830+0000 7fed3affd640 1 -- 172.21.15.6:0/1875239346 >> v1:172.21.15.6:6789/0 conn(0x7fed440b1b00 legacy=0x7fed441d7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:14.831 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.830+0000 7fed3affd640 10 monclient: _finish_hunting 0 2024-03-25T19:22:14.831 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.830+0000 7fed3affd640 1 monclient: found mon.b 2024-03-25T19:22:14.831 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.830+0000 7fed3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:14.831 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.830+0000 7fed3affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:22:14.831 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.830+0000 7fed3affd640 1 -- 172.21.15.6:0/1875239346 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed441d8830 con 0x7fed440b4e70 2024-03-25T19:22:14.832 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.830+0000 7fed3affd640 10 monclient: _finish_auth 0 2024-03-25T19:22:14.832 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.830+0000 7fed3affd640 10 monclient: _check_auth_tickets 2024-03-25T19:22:14.832 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.830+0000 7fed3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:14.832 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.830+0000 7fed4c316640 5 monclient: authenticate success, global_id 4600 2024-03-25T19:22:14.832 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.830+0000 7fed4c316640 10 monclient: _renew_subs 2024-03-25T19:22:14.832 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.830+0000 7fed4c316640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:22:14.832 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.830+0000 7fed4c316640 1 -- 172.21.15.6:0/1875239346 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed44136ed0 con 0x7fed440b4e70 2024-03-25T19:22:14.832 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.831+0000 7fed4c316640 10 client.4600.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:22:14.832 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.831+0000 7fed4c316640 10 monclient: _renew_subs 2024-03-25T19:22:14.832 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.831+0000 7fed4c316640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:22:14.832 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.831+0000 7fed4c316640 1 -- 172.21.15.6:0/1875239346 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fed44135c90 con 0x7fed440b4e70 2024-03-25T19:22:14.832 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.831+0000 7fed4c316640 1 librados: init done 2024-03-25T19:22:14.833 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.832+0000 7fed4c316640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:14.833 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.832+0000 7fed4c316640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:14.833 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.832+0000 7fed4c316640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:22:14.833 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:14.832+0000 7fed4c316640 1 -- 172.21.15.6:0/1875239346 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed441370c0 con 0x7fed440b4e70 2024-03-25T19:22:15.135 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:15.134+0000 7fed3affd640 1 -- 172.21.15.6:0/1875239346 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed3c001f20 con 0x7fed440b4e70 2024-03-25T19:22:15.135 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:15.134+0000 7fed3affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:15.135 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:15.134+0000 7fed3affd640 1 -- 172.21.15.6:0/1875239346 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fed3c004fb0 con 0x7fed440b4e70 2024-03-25T19:22:15.135 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:15.134+0000 7fed3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:15.135 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:15.134+0000 7fed488af640 4 set_mon_vals no callback set 2024-03-25T19:22:15.135 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:15.134+0000 7fed3affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:15.135 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:15.134+0000 7fed3affd640 10 monclient: dump: 2024-03-25T19:22:15.135 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:22:15.135 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:15.135 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:15.135 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:15.136 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:15.136 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:15.136 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:22:15.136 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:22:15.136 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:22:15.136 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:15.137 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:15.135+0000 7fed3affd640 1 -- 172.21.15.6:0/1875239346 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7fed3c01af80 con 0x7fed440b4e70 2024-03-25T19:22:15.137 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:15.136+0000 7fed4a0b2640 10 monclient: get_auth_request con 0x7fed2806e0f0 auth_method 0 2024-03-25T19:22:15.137 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:15.136+0000 7fed3affd640 10 client.4600.objecter ms_handle_connect 0x7fed2806e0f0 2024-03-25T19:22:15.141 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:15.141+0000 7fed3affd640 1 -- 172.21.15.6:0/1875239346 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7fed3c054a30 con 0x7fed440b4e70 2024-03-25T19:22:15.141 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:15.141+0000 7fed3affd640 10 client.4600.objecter ms_dispatch 0x7fed44000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:22:15.141 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:15.141+0000 7fed3affd640 3 client.4600.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:22:15.141 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:15.141+0000 7fed3affd640 3 client.4600.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:22:15.141 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:15.141+0000 7fed3affd640 20 client.4600.objecter dump_active .. 0 homeless 2024-03-25T19:22:15.141 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:15.141+0000 7fed3affd640 1 -- 172.21.15.6:0/1875239346 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fed3c050540 con 0x7fed440b4e70 2024-03-25T19:22:15.141 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:15.141+0000 7fed3affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:15.141 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:15.141+0000 7fed3affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:22:15.398 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:15.398+0000 7fed4c316640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:15.398 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:15.398+0000 7fed4c316640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:15.398 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:15.398+0000 7fed4c316640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:22:15.398 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:15.398+0000 7fed4c316640 1 -- 172.21.15.6:0/1875239346 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed441d7c00 con 0x7fed440b4e70 2024-03-25T19:22:15.441 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:15.441+0000 7fed3affd640 1 -- 172.21.15.6:0/1875239346 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7fed3c052470 con 0x7fed440b4e70 2024-03-25T19:22:15.442 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:15.441+0000 7fed3affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:15.442 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:15.441+0000 7fed3affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:22:15.442 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:22:15.442 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:22:15.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:15.443+0000 7fed4c316640 10 librados: watch_flush enter 2024-03-25T19:22:15.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:15.443+0000 7fed4c316640 10 librados: watch_flush exit 2024-03-25T19:22:15.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:15.443+0000 7fed4c316640 20 client.4600.objecter shutdown clearing up homeless session... 2024-03-25T19:22:15.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:15.443+0000 7fed4c316640 10 client.4600.objecter successfully canceled tick 2024-03-25T19:22:15.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:15.443+0000 7fed4c316640 1 -- 172.21.15.6:0/1875239346 >> v1:172.21.15.6:6800/2936652650 conn(0x7fed2806e0f0 legacy=0x7fed280705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:15.445 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:15.443+0000 7fed4c316640 10 monclient: shutdown 2024-03-25T19:22:15.445 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:15.443+0000 7fed4c316640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:15.445 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:15.443+0000 7fed4c316640 1 -- 172.21.15.6:0/1875239346 >> v1:172.21.15.192:6789/0 conn(0x7fed440b4e70 legacy=0x7fed441e2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:15.445 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:15.444+0000 7fed4c316640 1 -- 172.21.15.6:0/1875239346 shutdown_connections 2024-03-25T19:22:15.445 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:15.444+0000 7fed4c316640 1 -- 172.21.15.6:0/1875239346 >> 172.21.15.6:0/1875239346 conn(0x7fed440617a0 msgr2=0x7fed440b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:15.445 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:15.444+0000 7fed4c316640 1 -- 172.21.15.6:0/1875239346 shutdown_connections 2024-03-25T19:22:15.445 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:15.444+0000 7fed4c316640 1 -- 172.21.15.6:0/1875239346 wait complete. 2024-03-25T19:22:15.445 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:15.444+0000 7fed4c316640 1 librados: shutdown 2024-03-25T19:22:15.455 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:22:22.263 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:22:22.381 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.381+0000 7f1c4a50e640 10 monclient: get_monmap_and_config 2024-03-25T19:22:22.381 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.381+0000 7f1c4a50e640 10 monclient: build_initial_monmap 2024-03-25T19:22:22.381 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.381+0000 7f1c4a50e640 10 monclient: monmap: 2024-03-25T19:22:22.381 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:22:22.381 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:22.381 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:22:22.382441+0000 2024-03-25T19:22:22.381 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:22:22.382441+0000 2024-03-25T19:22:22.382 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:22.382 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:22.382 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:22:22.382 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:22:22.382 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:22:22.382 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:22.383 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.382+0000 7f1c4a50e640 1 Processor -- start 2024-03-25T19:22:22.383 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.382+0000 7f1c4a50e640 1 -- start start 2024-03-25T19:22:22.383 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.382+0000 7f1c4a50e640 10 monclient: init 2024-03-25T19:22:22.383 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.382+0000 7f1c4a50e640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:22.383 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.382+0000 7f1c4a50e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:22:22.383 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.382+0000 7f1c4a50e640 10 monclient(hunting): picked mon.noname-a con 0x7f1c440b4e70 addr v1:172.21.15.6:6789/0 2024-03-25T19:22:22.383 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.382+0000 7f1c4a50e640 10 monclient(hunting): picked mon.noname-c con 0x7f1c440b44b0 addr v1:172.21.15.192:6789/0 2024-03-25T19:22:22.383 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.383+0000 7f1c4a50e640 10 monclient(hunting): picked mon.noname-b con 0x7f1c440b1b00 addr v1:172.21.15.6:6790/0 2024-03-25T19:22:22.384 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.383+0000 7f1c4a50e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c440a96f0 con 0x7f1c440b4e70 2024-03-25T19:22:22.384 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.383+0000 7f1c4a50e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c440ab0e0 con 0x7f1c440b44b0 2024-03-25T19:22:22.384 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.383+0000 7f1c4a50e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c44145f70 con 0x7f1c440b1b00 2024-03-25T19:22:22.384 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.383+0000 7f1c4a50e640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:22.384 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.383+0000 7f1c4a50e640 10 monclient(hunting): authenticate will time out at 6446.622853s 2024-03-25T19:22:22.397 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.398+0000 7f1c43fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1c440b4e70 0x7f1c4413f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43948/0 (socket says 172.21.15.6:43948) 2024-03-25T19:22:22.398 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.398+0000 7f1c43fff640 1 -- 172.21.15.6:0/2060673897 learned_addr learned my addr 172.21.15.6:0/2060673897 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:22:22.420 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.420+0000 7f1c42ffd640 1 -- 172.21.15.6:0/2060673897 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 577006379 0 0) 0x7f1c44145f70 con 0x7f1c440b1b00 2024-03-25T19:22:22.420 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.420+0000 7f1c42ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:22:22.420 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.420+0000 7f1c42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:22.420 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.420+0000 7f1c42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:22.420 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.420+0000 7f1c42ffd640 10 monclient(hunting): my global_id is 4598 2024-03-25T19:22:22.420 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.420+0000 7f1c42ffd640 1 -- 172.21.15.6:0/2060673897 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c2c003520 con 0x7f1c440b1b00 2024-03-25T19:22:22.421 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.421+0000 7f1c42ffd640 1 -- 172.21.15.6:0/2060673897 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3250078666 0 0) 0x7f1c440ab0e0 con 0x7f1c440b44b0 2024-03-25T19:22:22.421 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.421+0000 7f1c42ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:22:22.422 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.421+0000 7f1c42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:22.422 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.421+0000 7f1c42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:22.422 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.421+0000 7f1c42ffd640 10 monclient(hunting): my global_id is 4603 2024-03-25T19:22:22.422 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.421+0000 7f1c42ffd640 1 -- 172.21.15.6:0/2060673897 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c44145f70 con 0x7f1c440b44b0 2024-03-25T19:22:22.425 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.425+0000 7f1c42ffd640 1 -- 172.21.15.6:0/2060673897 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 63020389 0 0) 0x7f1c440a96f0 con 0x7f1c440b4e70 2024-03-25T19:22:22.425 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.425+0000 7f1c42ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:22:22.425 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.425+0000 7f1c42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:22.425 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.425+0000 7f1c42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:22.425 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.425+0000 7f1c42ffd640 10 monclient(hunting): my global_id is 4590 2024-03-25T19:22:22.425 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.425+0000 7f1c42ffd640 1 -- 172.21.15.6:0/2060673897 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c440ab0e0 con 0x7f1c440b4e70 2024-03-25T19:22:22.433 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.432+0000 7f1c42ffd640 1 -- 172.21.15.6:0/2060673897 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3050465028 0 0) 0x7f1c44145f70 con 0x7f1c440b44b0 2024-03-25T19:22:22.433 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.432+0000 7f1c42ffd640 1 -- 172.21.15.6:0/2060673897 >> v1:172.21.15.6:6790/0 conn(0x7f1c440b1b00 legacy=0x7f1c4414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:22.433 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.432+0000 7f1c42ffd640 1 -- 172.21.15.6:0/2060673897 >> v1:172.21.15.6:6789/0 conn(0x7f1c440b4e70 legacy=0x7f1c4413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:22.433 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.432+0000 7f1c42ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:22:22.433 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.432+0000 7f1c42ffd640 1 monclient: found mon.noname-c 2024-03-25T19:22:22.433 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.432+0000 7f1c42ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:22.433 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.432+0000 7f1c42ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:22:22.434 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.432+0000 7f1c42ffd640 1 -- 172.21.15.6:0/2060673897 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c440a9b80 con 0x7f1c440b44b0 2024-03-25T19:22:22.434 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.432+0000 7f1c42ffd640 10 monclient: _finish_auth 0 2024-03-25T19:22:22.434 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.432+0000 7f1c42ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:22:22.434 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.432+0000 7f1c42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:22.434 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.432+0000 7f1c42ffd640 1 -- 172.21.15.6:0/2060673897 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f1c38001670 con 0x7f1c440b44b0 2024-03-25T19:22:22.434 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.432+0000 7f1c42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:22.434 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.432+0000 7f1c42ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:22:22.434 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.432+0000 7f1c42ffd640 10 monclient: dump: 2024-03-25T19:22:22.434 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:22:22.434 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:22.434 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:22.435 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:22.435 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:22.435 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:22.435 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:22:22.435 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:22:22.435 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:22:22.435 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:22.435 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.433+0000 7f1c4a50e640 5 monclient: authenticate success, global_id 4603 2024-03-25T19:22:22.435 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.433+0000 7f1c4a50e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:22.447 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.447+0000 7f1c42ffd640 1 -- 172.21.15.6:0/2060673897 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c380017d0 con 0x7f1c440b44b0 2024-03-25T19:22:22.447 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.447+0000 7f1c42ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:22.447 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.447+0000 7f1c42ffd640 1 -- 172.21.15.6:0/2060673897 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f1c38001be0 con 0x7f1c440b44b0 2024-03-25T19:22:22.447 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.447+0000 7f1c42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:22.447 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.447+0000 7f1c42ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:22.447 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.447+0000 7f1c42ffd640 10 monclient: dump: 2024-03-25T19:22:22.448 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:22:22.448 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:22.448 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:22.448 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:22.448 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:22.448 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:22.448 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:22:22.448 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:22:22.448 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:22:22.448 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:22.448 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.447+0000 7f1c4a50e640 10 monclient: get_monmap_and_config success 2024-03-25T19:22:22.448 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.447+0000 7f1c4a50e640 4 set_mon_vals no callback set 2024-03-25T19:22:22.450 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.448+0000 7f1c4a50e640 10 monclient: shutdown 2024-03-25T19:22:22.450 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.448+0000 7f1c4a50e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:22.450 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.448+0000 7f1c4a50e640 1 -- 172.21.15.6:0/2060673897 >> v1:172.21.15.192:6789/0 conn(0x7f1c440b44b0 legacy=0x7f1c44145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:22.450 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.448+0000 7f1c4a50e640 1 -- 172.21.15.6:0/2060673897 shutdown_connections 2024-03-25T19:22:22.450 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.448+0000 7f1c4a50e640 1 -- 172.21.15.6:0/2060673897 >> 172.21.15.6:0/2060673897 conn(0x7f1c440617a0 msgr2=0x7f1c44061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:22.450 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.448+0000 7f1c4a50e640 1 -- 172.21.15.6:0/2060673897 shutdown_connections 2024-03-25T19:22:22.450 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.448+0000 7f1c4a50e640 1 -- 172.21.15.6:0/2060673897 wait complete. 2024-03-25T19:22:22.450 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.449+0000 7f1c4a50e640 10 monclient: build_initial_monmap 2024-03-25T19:22:22.450 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.449+0000 7f1c4a50e640 10 monclient: monmap: 2024-03-25T19:22:22.451 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:22:22.451 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:22:22.451 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:22:22.451 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:22:22.451 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:22.451 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:22:22.451 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:22:22.451 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:22:22.451 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:22:22.451 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:22.451 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.450+0000 7f1c4a50e640 1 librados: starting msgr at 2024-03-25T19:22:22.451 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.450+0000 7f1c4a50e640 1 librados: starting objecter 2024-03-25T19:22:22.451 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.450+0000 7f1c4a50e640 1 Processor -- start 2024-03-25T19:22:22.451 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.450+0000 7f1c4a50e640 1 -- start start 2024-03-25T19:22:22.451 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.450+0000 7f1c4a50e640 1 librados: setting wanted keys 2024-03-25T19:22:22.452 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.450+0000 7f1c4a50e640 1 librados: calling monclient init 2024-03-25T19:22:22.452 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.450+0000 7f1c4a50e640 10 monclient: init 2024-03-25T19:22:22.452 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.451+0000 7f1c4a50e640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:22.453 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.451+0000 7f1c4a50e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:22:22.453 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.451+0000 7f1c4a50e640 10 monclient(hunting): picked mon.noname-c con 0x7f1c440b1b00 addr v1:172.21.15.6:6790/0 2024-03-25T19:22:22.453 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.451+0000 7f1c4a50e640 10 monclient(hunting): picked mon.noname-b con 0x7f1c440b44b0 addr v1:172.21.15.192:6789/0 2024-03-25T19:22:22.453 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.451+0000 7f1c4a50e640 10 monclient(hunting): picked mon.noname-a con 0x7f1c440b4e70 addr v1:172.21.15.6:6789/0 2024-03-25T19:22:22.453 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.451+0000 7f1c4a50e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c441376b0 con 0x7f1c440b4e70 2024-03-25T19:22:22.453 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.451+0000 7f1c4a50e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c44137990 con 0x7f1c440b44b0 2024-03-25T19:22:22.453 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.451+0000 7f1c4a50e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c44136780 con 0x7f1c440b1b00 2024-03-25T19:22:22.453 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.451+0000 7f1c4a50e640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:22.453 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.451+0000 7f1c4a50e640 10 monclient(hunting): authenticate will time out at 6446.691019s 2024-03-25T19:22:22.460 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.460+0000 7f1c48aab640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1c440b4e70 0x7f1c441e2020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44004/0 (socket says 172.21.15.6:44004) 2024-03-25T19:22:22.460 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.460+0000 7f1c48aab640 1 -- 172.21.15.6:0/3398797205 learned_addr learned my addr 172.21.15.6:0/3398797205 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:22:22.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.471+0000 7f1c40ff9640 10 client.?.objecter ms_handle_connect 0x7f1c440b4e70 2024-03-25T19:22:22.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.471+0000 7f1c40ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:22.472 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.472+0000 7f1c40ff9640 10 client.?.objecter ms_handle_connect 0x7f1c440b44b0 2024-03-25T19:22:22.472 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.472+0000 7f1c40ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:22.476 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.476+0000 7f1c40ff9640 10 client.?.objecter ms_handle_connect 0x7f1c440b1b00 2024-03-25T19:22:22.476 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.476+0000 7f1c40ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:22.476 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.476+0000 7f1c40ff9640 1 -- 172.21.15.6:0/3398797205 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1837856410 0 0) 0x7f1c441376b0 con 0x7f1c440b4e70 2024-03-25T19:22:22.476 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.476+0000 7f1c40ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:22:22.477 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.476+0000 7f1c40ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:22.477 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.476+0000 7f1c40ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:22.477 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.476+0000 7f1c40ff9640 10 monclient(hunting): my global_id is 4593 2024-03-25T19:22:22.477 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.476+0000 7f1c40ff9640 1 -- 172.21.15.6:0/3398797205 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c30004eb0 con 0x7f1c440b4e70 2024-03-25T19:22:22.481 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.481+0000 7f1c40ff9640 1 -- 172.21.15.6:0/3398797205 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2764842230 0 0) 0x7f1c44137990 con 0x7f1c440b44b0 2024-03-25T19:22:22.481 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.481+0000 7f1c40ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:22:22.481 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.481+0000 7f1c40ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:22.482 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.481+0000 7f1c40ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:22.482 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.481+0000 7f1c40ff9640 10 monclient(hunting): my global_id is 4606 2024-03-25T19:22:22.482 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.481+0000 7f1c40ff9640 1 -- 172.21.15.6:0/3398797205 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c441376b0 con 0x7f1c440b44b0 2024-03-25T19:22:22.488 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.488+0000 7f1c40ff9640 1 -- 172.21.15.6:0/3398797205 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 95134679 0 0) 0x7f1c44136780 con 0x7f1c440b1b00 2024-03-25T19:22:22.488 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.488+0000 7f1c40ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:22:22.489 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.488+0000 7f1c40ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:22.489 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.488+0000 7f1c40ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:22.489 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.488+0000 7f1c40ff9640 10 monclient(hunting): my global_id is 4601 2024-03-25T19:22:22.489 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.488+0000 7f1c40ff9640 1 -- 172.21.15.6:0/3398797205 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c44137990 con 0x7f1c440b1b00 2024-03-25T19:22:22.500 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.499+0000 7f1c40ff9640 1 -- 172.21.15.6:0/3398797205 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2507789551 0 0) 0x7f1c30004eb0 con 0x7f1c440b4e70 2024-03-25T19:22:22.500 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.499+0000 7f1c40ff9640 1 -- 172.21.15.6:0/3398797205 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c44136780 con 0x7f1c440b4e70 2024-03-25T19:22:22.500 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.499+0000 7f1c40ff9640 1 -- 172.21.15.6:0/3398797205 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f1c3c001cd0 con 0x7f1c440b4e70 2024-03-25T19:22:22.500 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.499+0000 7f1c40ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:22.500 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.499+0000 7f1c40ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:22:22.500 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.499+0000 7f1c40ff9640 10 monclient(hunting): dump: 2024-03-25T19:22:22.501 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:22:22.501 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:22.501 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:22.501 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:22.501 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:22.501 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:22.501 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:22:22.501 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:22:22.501 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:22:22.501 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:22.502 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.499+0000 7f1c40ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:22:22.502 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.502+0000 7f1c40ff9640 1 -- 172.21.15.6:0/3398797205 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4171096691 0 0) 0x7f1c441376b0 con 0x7f1c440b44b0 2024-03-25T19:22:22.502 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.502+0000 7f1c40ff9640 1 -- 172.21.15.6:0/3398797205 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c30004eb0 con 0x7f1c440b44b0 2024-03-25T19:22:22.502 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.502+0000 7f1c40ff9640 1 -- 172.21.15.6:0/3398797205 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f1c38001ae0 con 0x7f1c440b44b0 2024-03-25T19:22:22.503 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.502+0000 7f1c40ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:22.503 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.502+0000 7f1c40ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:22.503 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.502+0000 7f1c40ff9640 10 monclient(hunting): dump: 2024-03-25T19:22:22.503 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:22:22.503 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:22.503 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:22.503 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:22.503 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:22.504 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:22.504 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:22:22.504 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:22:22.504 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:22:22.504 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:22.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.505+0000 7f1c40ff9640 1 -- 172.21.15.6:0/3398797205 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2609598079 0 0) 0x7f1c44137990 con 0x7f1c440b1b00 2024-03-25T19:22:22.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.505+0000 7f1c40ff9640 1 -- 172.21.15.6:0/3398797205 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c441376b0 con 0x7f1c440b1b00 2024-03-25T19:22:22.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.505+0000 7f1c40ff9640 1 -- 172.21.15.6:0/3398797205 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f1c34001790 con 0x7f1c440b1b00 2024-03-25T19:22:22.507 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.505+0000 7f1c40ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:22.507 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.505+0000 7f1c40ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:22:22.507 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.505+0000 7f1c40ff9640 10 monclient(hunting): dump: 2024-03-25T19:22:22.507 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:22:22.507 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:22.507 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:22.507 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:22.507 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:22.507 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:22.508 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:22:22.508 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:22:22.508 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:22:22.508 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:22.513 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.512+0000 7f1c40ff9640 1 -- 172.21.15.6:0/3398797205 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1757160394 0 0) 0x7f1c44136780 con 0x7f1c440b4e70 2024-03-25T19:22:22.514 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.512+0000 7f1c40ff9640 1 -- 172.21.15.6:0/3398797205 >> v1:172.21.15.6:6790/0 conn(0x7f1c440b1b00 legacy=0x7f1c44138100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:22.514 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.512+0000 7f1c40ff9640 1 -- 172.21.15.6:0/3398797205 >> v1:172.21.15.192:6789/0 conn(0x7f1c440b44b0 legacy=0x7f1c44138810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:22.514 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.512+0000 7f1c40ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:22:22.514 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.512+0000 7f1c40ff9640 1 monclient: found mon.a 2024-03-25T19:22:22.514 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.512+0000 7f1c40ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:22.514 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.512+0000 7f1c40ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:22:22.514 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.512+0000 7f1c40ff9640 1 -- 172.21.15.6:0/3398797205 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c44135880 con 0x7f1c440b4e70 2024-03-25T19:22:22.515 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.512+0000 7f1c40ff9640 10 monclient: _finish_auth 0 2024-03-25T19:22:22.515 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.512+0000 7f1c40ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:22:22.515 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.512+0000 7f1c40ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:22.515 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.512+0000 7f1c4a50e640 5 monclient: authenticate success, global_id 4593 2024-03-25T19:22:22.515 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.512+0000 7f1c4a50e640 10 monclient: _renew_subs 2024-03-25T19:22:22.515 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.512+0000 7f1c4a50e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:22:22.515 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.512+0000 7f1c4a50e640 1 -- 172.21.15.6:0/3398797205 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c44135b30 con 0x7f1c440b4e70 2024-03-25T19:22:22.516 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.512+0000 7f1c4a50e640 10 client.4593.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:22:22.516 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.512+0000 7f1c4a50e640 10 monclient: _renew_subs 2024-03-25T19:22:22.516 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.512+0000 7f1c4a50e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:22:22.516 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.512+0000 7f1c4a50e640 1 -- 172.21.15.6:0/3398797205 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c44136200 con 0x7f1c440b4e70 2024-03-25T19:22:22.516 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.512+0000 7f1c4a50e640 1 librados: init done 2024-03-25T19:22:22.516 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.513+0000 7f1c4a50e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:22.516 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.513+0000 7f1c4a50e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:22.516 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.513+0000 7f1c4a50e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:22:22.517 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.513+0000 7f1c4a50e640 1 -- 172.21.15.6:0/3398797205 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c44136910 con 0x7f1c440b4e70 2024-03-25T19:22:22.863 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.863+0000 7f1c40ff9640 1 -- 172.21.15.6:0/3398797205 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c3c003990 con 0x7f1c440b4e70 2024-03-25T19:22:22.863 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.863+0000 7f1c40ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:22.863 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.863+0000 7f1c40ff9640 1 -- 172.21.15.6:0/3398797205 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f1c3c003d20 con 0x7f1c440b4e70 2024-03-25T19:22:22.863 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.863+0000 7f1c40ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:22.863 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.863+0000 7f1c40ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:22.863 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.863+0000 7f1c40ff9640 10 monclient: dump: 2024-03-25T19:22:22.864 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:22:22.864 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:22.864 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:22.864 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:22.864 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:22.864 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:22.864 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:22:22.864 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:22:22.864 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:22:22.864 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:22.864 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.863+0000 7f1c427fc640 4 set_mon_vals no callback set 2024-03-25T19:22:22.865 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.864+0000 7f1c40ff9640 1 -- 172.21.15.6:0/3398797205 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f1c3c003f40 con 0x7f1c440b4e70 2024-03-25T19:22:22.865 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.865+0000 7f1c43fff640 10 monclient: get_auth_request con 0x7f1c3006e0f0 auth_method 0 2024-03-25T19:22:22.865 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.865+0000 7f1c40ff9640 10 client.4593.objecter ms_handle_connect 0x7f1c3006e0f0 2024-03-25T19:22:22.872 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.872+0000 7f1c40ff9640 1 -- 172.21.15.6:0/3398797205 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f1c3c054a70 con 0x7f1c440b4e70 2024-03-25T19:22:22.872 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.872+0000 7f1c40ff9640 10 client.4593.objecter ms_dispatch 0x7f1c44000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:22:22.872 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.872+0000 7f1c40ff9640 3 client.4593.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:22:22.872 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.872+0000 7f1c40ff9640 3 client.4593.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:22:22.872 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.872+0000 7f1c40ff9640 20 client.4593.objecter dump_active .. 0 homeless 2024-03-25T19:22:22.872 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.872+0000 7f1c40ff9640 1 -- 172.21.15.6:0/3398797205 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1c3c050580 con 0x7f1c440b4e70 2024-03-25T19:22:22.872 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.872+0000 7f1c40ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:22.873 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:22.872+0000 7f1c40ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:22:23.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:23.116+0000 7f1c4a50e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:23.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:23.116+0000 7f1c4a50e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:23.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:23.116+0000 7f1c4a50e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:22:23.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:23.116+0000 7f1c4a50e640 1 -- 172.21.15.6:0/3398797205 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c4404f730 con 0x7f1c440b4e70 2024-03-25T19:22:23.188 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:23.188+0000 7f1c40ff9640 1 -- 172.21.15.6:0/3398797205 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f1c3c0524b0 con 0x7f1c440b4e70 2024-03-25T19:22:23.188 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:23.188+0000 7f1c40ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:23.188 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:23.188+0000 7f1c40ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:22:23.188 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:22:23.189 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:22:23.191 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:23.191+0000 7f1c4a50e640 10 librados: watch_flush enter 2024-03-25T19:22:23.191 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:23.191+0000 7f1c4a50e640 10 librados: watch_flush exit 2024-03-25T19:22:23.191 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:23.191+0000 7f1c4a50e640 20 client.4593.objecter shutdown clearing up homeless session... 2024-03-25T19:22:23.191 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:23.191+0000 7f1c4a50e640 10 client.4593.objecter successfully canceled tick 2024-03-25T19:22:23.191 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:23.191+0000 7f1c4a50e640 1 -- 172.21.15.6:0/3398797205 >> v1:172.21.15.6:6800/2936652650 conn(0x7f1c3006e0f0 legacy=0x7f1c300705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:23.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:23.191+0000 7f1c4a50e640 10 monclient: shutdown 2024-03-25T19:22:23.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:23.191+0000 7f1c4a50e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:23.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:23.191+0000 7f1c4a50e640 1 -- 172.21.15.6:0/3398797205 >> v1:172.21.15.6:6789/0 conn(0x7f1c440b4e70 legacy=0x7f1c441e2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:23.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:23.191+0000 7f1c4a50e640 1 -- 172.21.15.6:0/3398797205 shutdown_connections 2024-03-25T19:22:23.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:23.191+0000 7f1c4a50e640 1 -- 172.21.15.6:0/3398797205 >> 172.21.15.6:0/3398797205 conn(0x7f1c440617a0 msgr2=0x7f1c4413eb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:23.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:23.191+0000 7f1c4a50e640 1 -- 172.21.15.6:0/3398797205 shutdown_connections 2024-03-25T19:22:23.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:23.191+0000 7f1c4a50e640 1 -- 172.21.15.6:0/3398797205 wait complete. 2024-03-25T19:22:23.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:23.191+0000 7f1c4a50e640 1 librados: shutdown 2024-03-25T19:22:23.203 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:22:29.098 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:08:46.445 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:29.105 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:08:46.451 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:29.146 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:08:46.493 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:29.160 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:08:46.478 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:29.171 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:08:46.518 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:29.171 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:08:46.489 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:29.179 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:08:46.525 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:29.198 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:08:46.516 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:29.208 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:08:46.555 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:30.010 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:22:30.126 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.126+0000 7f750f22e640 10 monclient: get_monmap_and_config 2024-03-25T19:22:30.126 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.126+0000 7f750f22e640 10 monclient: build_initial_monmap 2024-03-25T19:22:30.126 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.126+0000 7f750f22e640 10 monclient: monmap: 2024-03-25T19:22:30.126 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:22:30.126 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:30.127 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:22:30.127320+0000 2024-03-25T19:22:30.127 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:22:30.127320+0000 2024-03-25T19:22:30.127 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:30.127 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:30.127 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:22:30.127 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:22:30.127 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:22:30.127 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:30.128 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.127+0000 7f750f22e640 1 Processor -- start 2024-03-25T19:22:30.128 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.127+0000 7f750f22e640 1 -- start start 2024-03-25T19:22:30.128 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.127+0000 7f750f22e640 10 monclient: init 2024-03-25T19:22:30.128 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.127+0000 7f750f22e640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:30.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.127+0000 7f750f22e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:22:30.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.127+0000 7f750f22e640 10 monclient(hunting): picked mon.noname-c con 0x7f75080b49f0 addr v1:172.21.15.192:6789/0 2024-03-25T19:22:30.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.127+0000 7f750f22e640 10 monclient(hunting): picked mon.noname-a con 0x7f75080b4dd0 addr v1:172.21.15.6:6789/0 2024-03-25T19:22:30.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.127+0000 7f750f22e640 10 monclient(hunting): picked mon.noname-b con 0x7f75080a9a90 addr v1:172.21.15.6:6790/0 2024-03-25T19:22:30.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.127+0000 7f750f22e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75080aa070 con 0x7f75080b4dd0 2024-03-25T19:22:30.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.127+0000 7f750f22e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75080aaf40 con 0x7f75080b49f0 2024-03-25T19:22:30.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.127+0000 7f750f22e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75080ab220 con 0x7f75080a9a90 2024-03-25T19:22:30.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.127+0000 7f750f22e640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:30.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.127+0000 7f750f22e640 10 monclient(hunting): authenticate will time out at 6454.367620s 2024-03-25T19:22:30.162 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.162+0000 7f750cfca640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f75080b49f0 0x7f750813bd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:60322/0 (socket says 172.21.15.6:60322) 2024-03-25T19:22:30.162 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.162+0000 7f750cfca640 1 -- 172.21.15.6:0/1459616841 learned_addr learned my addr 172.21.15.6:0/1459616841 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:22:30.180 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.180+0000 7f75077fe640 1 -- 172.21.15.6:0/1459616841 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 171200817 0 0) 0x7f75080aaf40 con 0x7f75080b49f0 2024-03-25T19:22:30.180 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.180+0000 7f75077fe640 10 monclient(hunting): _negotiate 2024-03-25T19:22:30.180 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.180+0000 7f75077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:30.180 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.180+0000 7f75077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:30.181 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.180+0000 7f75077fe640 10 monclient(hunting): my global_id is 4609 2024-03-25T19:22:30.181 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.180+0000 7f75077fe640 1 -- 172.21.15.6:0/1459616841 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74f8003520 con 0x7f75080b49f0 2024-03-25T19:22:30.185 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.185+0000 7f75077fe640 1 -- 172.21.15.6:0/1459616841 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3353779736 0 0) 0x7f75080ab220 con 0x7f75080a9a90 2024-03-25T19:22:30.185 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.185+0000 7f75077fe640 10 monclient(hunting): _negotiate 2024-03-25T19:22:30.186 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.185+0000 7f75077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:30.186 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.185+0000 7f75077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:30.186 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.185+0000 7f75077fe640 10 monclient(hunting): my global_id is 4604 2024-03-25T19:22:30.186 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.185+0000 7f75077fe640 1 -- 172.21.15.6:0/1459616841 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75080aaf40 con 0x7f75080a9a90 2024-03-25T19:22:30.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.191+0000 7f75077fe640 1 -- 172.21.15.6:0/1459616841 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3502808739 0 0) 0x7f74f8003520 con 0x7f75080b49f0 2024-03-25T19:22:30.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.191+0000 7f75077fe640 1 -- 172.21.15.6:0/1459616841 >> v1:172.21.15.6:6790/0 conn(0x7f75080a9a90 legacy=0x7f7508142c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:30.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.191+0000 7f75077fe640 1 -- 172.21.15.6:0/1459616841 >> v1:172.21.15.6:6789/0 conn(0x7f75080b4dd0 legacy=0x7f7508142520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:30.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.191+0000 7f75077fe640 10 monclient: _finish_hunting 0 2024-03-25T19:22:30.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.191+0000 7f75077fe640 1 monclient: found mon.noname-c 2024-03-25T19:22:30.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.192+0000 7f75077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:30.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.192+0000 7f75077fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:22:30.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.192+0000 7f75077fe640 1 -- 172.21.15.6:0/1459616841 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f750813c410 con 0x7f75080b49f0 2024-03-25T19:22:30.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.192+0000 7f75077fe640 10 monclient: _finish_auth 0 2024-03-25T19:22:30.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.192+0000 7f75077fe640 10 monclient: _check_auth_tickets 2024-03-25T19:22:30.193 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.192+0000 7f75077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:30.193 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.192+0000 7f75077fe640 1 -- 172.21.15.6:0/1459616841 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f74fc001690 con 0x7f75080b49f0 2024-03-25T19:22:30.193 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.192+0000 7f75077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:30.193 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.192+0000 7f75077fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:22:30.193 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.192+0000 7f75077fe640 10 monclient: dump: 2024-03-25T19:22:30.193 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:22:30.193 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:30.193 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:30.193 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:30.193 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:30.193 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:30.193 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:22:30.193 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:22:30.193 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:22:30.193 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:30.194 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.192+0000 7f750f22e640 5 monclient: authenticate success, global_id 4609 2024-03-25T19:22:30.194 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.192+0000 7f750f22e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:30.208 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.209+0000 7f75077fe640 1 -- 172.21.15.6:0/1459616841 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74fc0017f0 con 0x7f75080b49f0 2024-03-25T19:22:30.209 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.209+0000 7f75077fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:30.209 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.209+0000 7f75077fe640 1 -- 172.21.15.6:0/1459616841 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f74fc001c00 con 0x7f75080b49f0 2024-03-25T19:22:30.209 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.209+0000 7f750f22e640 10 monclient: get_monmap_and_config success 2024-03-25T19:22:30.209 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.209+0000 7f750f22e640 4 set_mon_vals no callback set 2024-03-25T19:22:30.209 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.209+0000 7f75077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:30.209 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.209+0000 7f75077fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:30.209 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.209+0000 7f75077fe640 10 monclient: dump: 2024-03-25T19:22:30.209 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:22:30.209 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:30.209 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:30.209 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:30.209 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:30.210 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:30.210 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:22:30.210 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:22:30.210 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:22:30.210 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:30.212 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.209+0000 7f750f22e640 10 monclient: shutdown 2024-03-25T19:22:30.213 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.209+0000 7f750f22e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:30.213 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.209+0000 7f750f22e640 1 -- 172.21.15.6:0/1459616841 >> v1:172.21.15.192:6789/0 conn(0x7f75080b49f0 legacy=0x7f750813bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:30.213 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.209+0000 7f750f22e640 1 -- 172.21.15.6:0/1459616841 shutdown_connections 2024-03-25T19:22:30.213 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.209+0000 7f750f22e640 1 -- 172.21.15.6:0/1459616841 >> 172.21.15.6:0/1459616841 conn(0x7f75080615c0 msgr2=0x7f75080619a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:30.213 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.209+0000 7f750f22e640 1 -- 172.21.15.6:0/1459616841 shutdown_connections 2024-03-25T19:22:30.213 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.209+0000 7f750f22e640 1 -- 172.21.15.6:0/1459616841 wait complete. 2024-03-25T19:22:30.213 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.210+0000 7f750f22e640 10 monclient: build_initial_monmap 2024-03-25T19:22:30.213 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.210+0000 7f750f22e640 10 monclient: monmap: 2024-03-25T19:22:30.213 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:22:30.213 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:22:30.213 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:22:30.213 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:22:30.213 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:30.213 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:22:30.214 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:22:30.214 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:22:30.214 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:22:30.214 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:30.214 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.211+0000 7f750f22e640 1 librados: starting msgr at 2024-03-25T19:22:30.214 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.211+0000 7f750f22e640 1 librados: starting objecter 2024-03-25T19:22:30.214 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.211+0000 7f750f22e640 1 Processor -- start 2024-03-25T19:22:30.214 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.211+0000 7f750f22e640 1 -- start start 2024-03-25T19:22:30.214 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.211+0000 7f750f22e640 1 librados: setting wanted keys 2024-03-25T19:22:30.214 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.211+0000 7f750f22e640 1 librados: calling monclient init 2024-03-25T19:22:30.214 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.211+0000 7f750f22e640 10 monclient: init 2024-03-25T19:22:30.214 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.211+0000 7f750f22e640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:30.214 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.212+0000 7f750f22e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:22:30.215 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.212+0000 7f750f22e640 10 monclient(hunting): picked mon.noname-c con 0x7f75080a9a90 addr v1:172.21.15.6:6790/0 2024-03-25T19:22:30.215 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.212+0000 7f750f22e640 10 monclient(hunting): picked mon.noname-b con 0x7f75080b49f0 addr v1:172.21.15.192:6789/0 2024-03-25T19:22:30.215 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.212+0000 7f750f22e640 10 monclient(hunting): picked mon.noname-a con 0x7f75080b4dd0 addr v1:172.21.15.6:6789/0 2024-03-25T19:22:30.215 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.212+0000 7f750f22e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f750813c7a0 con 0x7f75080b4dd0 2024-03-25T19:22:30.215 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.212+0000 7f750f22e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75081dbe40 con 0x7f75080b49f0 2024-03-25T19:22:30.215 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.212+0000 7f750f22e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f750813abe0 con 0x7f75080a9a90 2024-03-25T19:22:30.215 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.212+0000 7f750f22e640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:30.215 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.212+0000 7f750f22e640 10 monclient(hunting): authenticate will time out at 6454.451972s 2024-03-25T19:22:30.218 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.219+0000 7f7507fff640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f75080b49f0 0x7f75081db020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:60358/0 (socket says 172.21.15.6:60358) 2024-03-25T19:22:30.218 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.219+0000 7f7507fff640 1 -- 172.21.15.6:0/243617089 learned_addr learned my addr 172.21.15.6:0/243617089 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:22:30.220 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.220+0000 7f75057fa640 10 client.?.objecter ms_handle_connect 0x7f75080b49f0 2024-03-25T19:22:30.220 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.220+0000 7f75057fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:30.224 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.224+0000 7f75057fa640 10 client.?.objecter ms_handle_connect 0x7f75080b4dd0 2024-03-25T19:22:30.224 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.224+0000 7f75057fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:30.227 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.227+0000 7f75057fa640 10 client.?.objecter ms_handle_connect 0x7f75080a9a90 2024-03-25T19:22:30.227 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.227+0000 7f75057fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:30.236 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.236+0000 7f75057fa640 1 -- 172.21.15.6:0/243617089 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3921207762 0 0) 0x7f750813c7a0 con 0x7f75080b4dd0 2024-03-25T19:22:30.236 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.236+0000 7f75057fa640 10 monclient(hunting): _negotiate 2024-03-25T19:22:30.236 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.236+0000 7f75057fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:30.236 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.236+0000 7f75057fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:30.236 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.236+0000 7f75057fa640 10 monclient(hunting): my global_id is 4599 2024-03-25T19:22:30.237 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.236+0000 7f75057fa640 1 -- 172.21.15.6:0/243617089 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74ec004eb0 con 0x7f75080b4dd0 2024-03-25T19:22:30.268 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.268+0000 7f75057fa640 1 -- 172.21.15.6:0/243617089 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1793690444 0 0) 0x7f74ec004eb0 con 0x7f75080b4dd0 2024-03-25T19:22:30.269 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.268+0000 7f75057fa640 1 -- 172.21.15.6:0/243617089 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f750813c7a0 con 0x7f75080b4dd0 2024-03-25T19:22:30.269 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.268+0000 7f75057fa640 1 -- 172.21.15.6:0/243617089 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f7500001bd0 con 0x7f75080b4dd0 2024-03-25T19:22:30.269 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.268+0000 7f75057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:30.269 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.268+0000 7f75057fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:22:30.269 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.268+0000 7f75057fa640 10 monclient(hunting): dump: 2024-03-25T19:22:30.269 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:22:30.269 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:30.269 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:30.269 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:30.269 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:30.269 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:30.270 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:22:30.270 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:22:30.270 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:22:30.270 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:30.270 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.269+0000 7f75057fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:22:30.325 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.325+0000 7f75057fa640 1 -- 172.21.15.6:0/243617089 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4235367746 0 0) 0x7f750813c7a0 con 0x7f75080b4dd0 2024-03-25T19:22:30.325 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.325+0000 7f75057fa640 1 -- 172.21.15.6:0/243617089 >> v1:172.21.15.6:6790/0 conn(0x7f75080a9a90 legacy=0x7f7508137860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:30.325 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.325+0000 7f75057fa640 1 -- 172.21.15.6:0/243617089 >> v1:172.21.15.192:6789/0 conn(0x7f75080b49f0 legacy=0x7f75081db020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:30.325 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.325+0000 7f75057fa640 10 monclient: _finish_hunting 0 2024-03-25T19:22:30.325 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.325+0000 7f75057fa640 1 monclient: found mon.a 2024-03-25T19:22:30.325 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.325+0000 7f75057fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:30.325 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.325+0000 7f75057fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:22:30.326 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.325+0000 7f75057fa640 1 -- 172.21.15.6:0/243617089 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f750813aec0 con 0x7f75080b4dd0 2024-03-25T19:22:30.326 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.325+0000 7f75057fa640 10 monclient: _finish_auth 0 2024-03-25T19:22:30.326 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.325+0000 7f75057fa640 10 monclient: _check_auth_tickets 2024-03-25T19:22:30.326 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.325+0000 7f75057fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:30.326 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.325+0000 7f750f22e640 5 monclient: authenticate success, global_id 4599 2024-03-25T19:22:30.326 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.325+0000 7f750f22e640 10 monclient: _renew_subs 2024-03-25T19:22:30.326 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.325+0000 7f750f22e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:22:30.326 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.325+0000 7f750f22e640 1 -- 172.21.15.6:0/243617089 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f750813b1a0 con 0x7f75080b4dd0 2024-03-25T19:22:30.326 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.325+0000 7f750f22e640 10 client.4599.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:22:30.326 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.325+0000 7f750f22e640 10 monclient: _renew_subs 2024-03-25T19:22:30.326 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.325+0000 7f750f22e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:22:30.326 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.325+0000 7f750f22e640 1 -- 172.21.15.6:0/243617089 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f750813c0c0 con 0x7f75080b4dd0 2024-03-25T19:22:30.326 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.325+0000 7f750f22e640 1 librados: init done 2024-03-25T19:22:30.327 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.326+0000 7f750f22e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:30.327 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.326+0000 7f750f22e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:30.327 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.326+0000 7f750f22e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:22:30.327 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.326+0000 7f750f22e640 1 -- 172.21.15.6:0/243617089 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75081417f0 con 0x7f75080b4dd0 2024-03-25T19:22:30.649 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.649+0000 7f75057fa640 1 -- 172.21.15.6:0/243617089 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7500003990 con 0x7f75080b4dd0 2024-03-25T19:22:30.649 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.649+0000 7f75057fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:30.649 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.649+0000 7f75057fa640 1 -- 172.21.15.6:0/243617089 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f7500002100 con 0x7f75080b4dd0 2024-03-25T19:22:30.649 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.649+0000 7f75057fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:30.649 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.649+0000 7f75057fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:30.649 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.649+0000 7f75057fa640 10 monclient: dump: 2024-03-25T19:22:30.649 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:22:30.650 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:30.650 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:30.650 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:30.650 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:30.650 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:30.650 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:22:30.650 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:22:30.650 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:22:30.650 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:30.650 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.649+0000 7f7506ffd640 4 set_mon_vals no callback set 2024-03-25T19:22:30.651 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.649+0000 7f75057fa640 1 -- 172.21.15.6:0/243617089 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f7500019b90 con 0x7f75080b4dd0 2024-03-25T19:22:30.651 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.650+0000 7f750cfca640 10 monclient: get_auth_request con 0x7f74ec06dc90 auth_method 0 2024-03-25T19:22:30.651 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.650+0000 7f75057fa640 10 client.4599.objecter ms_handle_connect 0x7f74ec06dc90 2024-03-25T19:22:30.656 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.656+0000 7f75057fa640 1 -- 172.21.15.6:0/243617089 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f7500053910 con 0x7f75080b4dd0 2024-03-25T19:22:30.656 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.656+0000 7f75057fa640 10 client.4599.objecter ms_dispatch 0x7f7508000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:22:30.656 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.656+0000 7f75057fa640 3 client.4599.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:22:30.656 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.656+0000 7f75057fa640 3 client.4599.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:22:30.656 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.656+0000 7f75057fa640 20 client.4599.objecter dump_active .. 0 homeless 2024-03-25T19:22:30.656 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.656+0000 7f75057fa640 1 -- 172.21.15.6:0/243617089 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f750004f250 con 0x7f75080b4dd0 2024-03-25T19:22:30.656 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.656+0000 7f75057fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:30.656 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.656+0000 7f75057fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:22:30.915 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.915+0000 7f750f22e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:30.916 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.915+0000 7f750f22e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:30.916 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.915+0000 7f750f22e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:22:30.916 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.915+0000 7f750f22e640 1 -- 172.21.15.6:0/243617089 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f750804f730 con 0x7f75080b4dd0 2024-03-25T19:22:30.955 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.955+0000 7f75057fa640 1 -- 172.21.15.6:0/243617089 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f7500051350 con 0x7f75080b4dd0 2024-03-25T19:22:30.955 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.955+0000 7f75057fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:30.955 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.955+0000 7f75057fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:22:30.956 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:22:30.956 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:22:30.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.957+0000 7f750f22e640 10 librados: watch_flush enter 2024-03-25T19:22:30.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.957+0000 7f750f22e640 10 librados: watch_flush exit 2024-03-25T19:22:30.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.957+0000 7f750f22e640 20 client.4599.objecter shutdown clearing up homeless session... 2024-03-25T19:22:30.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.957+0000 7f750f22e640 10 client.4599.objecter successfully canceled tick 2024-03-25T19:22:30.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.957+0000 7f750f22e640 1 -- 172.21.15.6:0/243617089 >> v1:172.21.15.6:6800/2936652650 conn(0x7f74ec06dc90 legacy=0x7f74ec070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:30.959 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.958+0000 7f750f22e640 10 monclient: shutdown 2024-03-25T19:22:30.959 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.958+0000 7f750f22e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:30.959 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.958+0000 7f750f22e640 1 -- 172.21.15.6:0/243617089 >> v1:172.21.15.6:6789/0 conn(0x7f75080b4dd0 legacy=0x7f75081db730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:30.959 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.958+0000 7f750f22e640 1 -- 172.21.15.6:0/243617089 shutdown_connections 2024-03-25T19:22:30.959 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.958+0000 7f750f22e640 1 -- 172.21.15.6:0/243617089 >> 172.21.15.6:0/243617089 conn(0x7f75080615c0 msgr2=0x7f7508142c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:30.959 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.958+0000 7f750f22e640 1 -- 172.21.15.6:0/243617089 shutdown_connections 2024-03-25T19:22:30.959 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.958+0000 7f750f22e640 1 -- 172.21.15.6:0/243617089 wait complete. 2024-03-25T19:22:30.959 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:30.958+0000 7f750f22e640 1 librados: shutdown 2024-03-25T19:22:30.968 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:22:37.776 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:22:37.887 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.887+0000 7ff95cf97640 10 monclient: get_monmap_and_config 2024-03-25T19:22:37.887 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.887+0000 7ff95cf97640 10 monclient: build_initial_monmap 2024-03-25T19:22:37.887 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.887+0000 7ff95cf97640 10 monclient: monmap: 2024-03-25T19:22:37.888 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:22:37.888 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:37.888 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:22:37.888595+0000 2024-03-25T19:22:37.888 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:22:37.888595+0000 2024-03-25T19:22:37.888 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:37.888 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:37.888 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:22:37.888 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:22:37.888 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:22:37.888 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:37.889 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.888+0000 7ff95cf97640 1 Processor -- start 2024-03-25T19:22:37.889 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.888+0000 7ff95cf97640 1 -- start start 2024-03-25T19:22:37.889 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.888+0000 7ff95cf97640 10 monclient: init 2024-03-25T19:22:37.889 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.888+0000 7ff95cf97640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:37.889 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.888+0000 7ff95cf97640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:22:37.890 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.889+0000 7ff95cf97640 10 monclient(hunting): picked mon.noname-c con 0x7ff9580b4e70 addr v1:172.21.15.192:6789/0 2024-03-25T19:22:37.890 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.889+0000 7ff95cf97640 10 monclient(hunting): picked mon.noname-b con 0x7ff9580b44b0 addr v1:172.21.15.6:6790/0 2024-03-25T19:22:37.890 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.889+0000 7ff95cf97640 10 monclient(hunting): picked mon.noname-a con 0x7ff9580b1b00 addr v1:172.21.15.6:6789/0 2024-03-25T19:22:37.890 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.889+0000 7ff95cf97640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9580a96f0 con 0x7ff9580b1b00 2024-03-25T19:22:37.890 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.889+0000 7ff95cf97640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9580ab0e0 con 0x7ff9580b4e70 2024-03-25T19:22:37.890 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.889+0000 7ff95cf97640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff958145f70 con 0x7ff9580b44b0 2024-03-25T19:22:37.890 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.889+0000 7ff95cf97640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:37.890 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.889+0000 7ff95cf97640 10 monclient(hunting): authenticate will time out at 6462.128846s 2024-03-25T19:22:37.890 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.889+0000 7ff94ffff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7ff9580b44b0 0x7ff958145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:40090/0 (socket says 172.21.15.6:40090) 2024-03-25T19:22:37.890 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.889+0000 7ff94ffff640 1 -- 172.21.15.6:0/1759772509 learned_addr learned my addr 172.21.15.6:0/1759772509 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:22:37.907 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.907+0000 7ff955d9b640 1 -- 172.21.15.6:0/1759772509 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 78929339 0 0) 0x7ff9580a96f0 con 0x7ff9580b1b00 2024-03-25T19:22:37.907 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.907+0000 7ff955d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:22:37.907 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.907+0000 7ff955d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:37.907 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.907+0000 7ff955d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:37.907 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.907+0000 7ff955d9b640 10 monclient(hunting): my global_id is 4602 2024-03-25T19:22:37.907 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.907+0000 7ff955d9b640 1 -- 172.21.15.6:0/1759772509 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff944003520 con 0x7ff9580b1b00 2024-03-25T19:22:37.946 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.946+0000 7ff955d9b640 1 -- 172.21.15.6:0/1759772509 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 11508866 0 0) 0x7ff9580ab0e0 con 0x7ff9580b4e70 2024-03-25T19:22:37.946 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.946+0000 7ff955d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:22:37.947 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.946+0000 7ff955d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:37.947 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.946+0000 7ff955d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:37.947 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.947+0000 7ff955d9b640 10 monclient(hunting): my global_id is 4615 2024-03-25T19:22:37.947 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.947+0000 7ff955d9b640 1 -- 172.21.15.6:0/1759772509 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9580a96f0 con 0x7ff9580b4e70 2024-03-25T19:22:37.951 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.950+0000 7ff955d9b640 1 -- 172.21.15.6:0/1759772509 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1481418405 0 0) 0x7ff944003520 con 0x7ff9580b1b00 2024-03-25T19:22:37.951 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.950+0000 7ff955d9b640 1 -- 172.21.15.6:0/1759772509 >> v1:172.21.15.6:6790/0 conn(0x7ff9580b44b0 legacy=0x7ff958145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:37.951 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.950+0000 7ff955d9b640 1 -- 172.21.15.6:0/1759772509 >> v1:172.21.15.192:6789/0 conn(0x7ff9580b4e70 legacy=0x7ff95813f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:37.951 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.950+0000 7ff955d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:22:37.951 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.950+0000 7ff955d9b640 1 monclient: found mon.noname-a 2024-03-25T19:22:37.951 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.950+0000 7ff955d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:37.951 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.950+0000 7ff955d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-03-25T19:22:37.951 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.950+0000 7ff955d9b640 1 -- 172.21.15.6:0/1759772509 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9580a9b80 con 0x7ff9580b1b00 2024-03-25T19:22:37.951 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.950+0000 7ff955d9b640 10 monclient: _finish_auth 0 2024-03-25T19:22:37.952 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.950+0000 7ff955d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:22:37.952 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.950+0000 7ff955d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:37.952 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.950+0000 7ff955d9b640 1 -- 172.21.15.6:0/1759772509 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7ff950001a10 con 0x7ff9580b1b00 2024-03-25T19:22:37.952 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.950+0000 7ff955d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:37.952 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.950+0000 7ff955d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:22:37.952 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.950+0000 7ff955d9b640 10 monclient: dump: 2024-03-25T19:22:37.952 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:22:37.952 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:37.952 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:37.952 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:37.952 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:37.952 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:37.952 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:22:37.952 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:22:37.953 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:22:37.953 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:37.953 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.950+0000 7ff95cf97640 5 monclient: authenticate success, global_id 4602 2024-03-25T19:22:37.953 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.950+0000 7ff95cf97640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:37.976 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.976+0000 7ff955d9b640 1 -- 172.21.15.6:0/1759772509 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff950001b70 con 0x7ff9580b1b00 2024-03-25T19:22:37.976 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.976+0000 7ff955d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:37.976 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.976+0000 7ff955d9b640 1 -- 172.21.15.6:0/1759772509 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7ff950001f80 con 0x7ff9580b1b00 2024-03-25T19:22:37.976 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.976+0000 7ff955d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:37.976 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.976+0000 7ff955d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:37.976 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.976+0000 7ff955d9b640 10 monclient: dump: 2024-03-25T19:22:37.976 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:22:37.976 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:37.976 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:37.976 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:37.977 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:37.977 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:37.977 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:22:37.977 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:22:37.977 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:22:37.977 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:37.977 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.976+0000 7ff95cf97640 10 monclient: get_monmap_and_config success 2024-03-25T19:22:37.977 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.976+0000 7ff95cf97640 4 set_mon_vals no callback set 2024-03-25T19:22:37.978 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.977+0000 7ff95cf97640 10 monclient: shutdown 2024-03-25T19:22:37.979 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.977+0000 7ff95cf97640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:37.979 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.977+0000 7ff95cf97640 1 -- 172.21.15.6:0/1759772509 >> v1:172.21.15.6:6789/0 conn(0x7ff9580b1b00 legacy=0x7ff95814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:37.979 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.977+0000 7ff95cf97640 1 -- 172.21.15.6:0/1759772509 shutdown_connections 2024-03-25T19:22:37.979 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.977+0000 7ff95cf97640 1 -- 172.21.15.6:0/1759772509 >> 172.21.15.6:0/1759772509 conn(0x7ff9580617a0 msgr2=0x7ff958061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:37.979 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.977+0000 7ff95cf97640 1 -- 172.21.15.6:0/1759772509 shutdown_connections 2024-03-25T19:22:37.979 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.977+0000 7ff95cf97640 1 -- 172.21.15.6:0/1759772509 wait complete. 2024-03-25T19:22:37.979 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.978+0000 7ff95cf97640 10 monclient: build_initial_monmap 2024-03-25T19:22:37.979 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.978+0000 7ff95cf97640 10 monclient: monmap: 2024-03-25T19:22:37.979 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:22:37.979 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:22:37.979 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:22:37.979 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:22:37.979 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:37.979 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:22:37.980 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:22:37.980 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:22:37.980 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:22:37.980 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:37.980 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.978+0000 7ff95cf97640 1 librados: starting msgr at 2024-03-25T19:22:37.980 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.978+0000 7ff95cf97640 1 librados: starting objecter 2024-03-25T19:22:37.980 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.978+0000 7ff95cf97640 1 Processor -- start 2024-03-25T19:22:37.980 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.979+0000 7ff95cf97640 1 -- start start 2024-03-25T19:22:37.980 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.979+0000 7ff95cf97640 1 librados: setting wanted keys 2024-03-25T19:22:37.980 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.979+0000 7ff95cf97640 1 librados: calling monclient init 2024-03-25T19:22:37.980 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.979+0000 7ff95cf97640 10 monclient: init 2024-03-25T19:22:37.981 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.979+0000 7ff95cf97640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:37.981 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.979+0000 7ff95cf97640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:22:37.981 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.979+0000 7ff95cf97640 10 monclient(hunting): picked mon.noname-b con 0x7ff9580b1b00 addr v1:172.21.15.192:6789/0 2024-03-25T19:22:37.982 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.979+0000 7ff95cf97640 10 monclient(hunting): picked mon.noname-c con 0x7ff9580b44b0 addr v1:172.21.15.6:6790/0 2024-03-25T19:22:37.982 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.979+0000 7ff95cf97640 10 monclient(hunting): picked mon.noname-a con 0x7ff9580b4e70 addr v1:172.21.15.6:6789/0 2024-03-25T19:22:37.982 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.979+0000 7ff95cf97640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff958142d40 con 0x7ff9580b4e70 2024-03-25T19:22:37.982 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.979+0000 7ff95cf97640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9581438e0 con 0x7ff9580b1b00 2024-03-25T19:22:37.982 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.980+0000 7ff95cf97640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff958131920 con 0x7ff9580b44b0 2024-03-25T19:22:37.982 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.980+0000 7ff95cf97640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:37.982 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.980+0000 7ff95cf97640 10 monclient(hunting): authenticate will time out at 6462.219809s 2024-03-25T19:22:37.984 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.985+0000 7ff95659c640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7ff9580b1b00 0x7ff9581431d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:36746/0 (socket says 172.21.15.6:36746) 2024-03-25T19:22:37.984 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.985+0000 7ff95659c640 1 -- 172.21.15.6:0/829124202 learned_addr learned my addr 172.21.15.6:0/829124202 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:22:37.985 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.985+0000 7ff94effd640 10 client.?.objecter ms_handle_connect 0x7ff9580b1b00 2024-03-25T19:22:37.985 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.985+0000 7ff94effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:37.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.988+0000 7ff94effd640 10 client.?.objecter ms_handle_connect 0x7ff9580b4e70 2024-03-25T19:22:37.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.988+0000 7ff94effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:37.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.989+0000 7ff94effd640 10 client.?.objecter ms_handle_connect 0x7ff9580b44b0 2024-03-25T19:22:37.989 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.989+0000 7ff94effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:37.990 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.990+0000 7ff94effd640 1 -- 172.21.15.6:0/829124202 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2874940109 0 0) 0x7ff9581438e0 con 0x7ff9580b1b00 2024-03-25T19:22:37.990 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.990+0000 7ff94effd640 10 monclient(hunting): _negotiate 2024-03-25T19:22:37.990 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.990+0000 7ff94effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:37.990 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.990+0000 7ff94effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:37.990 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.990+0000 7ff94effd640 10 monclient(hunting): my global_id is 4618 2024-03-25T19:22:37.990 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.990+0000 7ff94effd640 1 -- 172.21.15.6:0/829124202 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff938004eb0 con 0x7ff9580b1b00 2024-03-25T19:22:37.994 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.994+0000 7ff94effd640 1 -- 172.21.15.6:0/829124202 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1972481003 0 0) 0x7ff958142d40 con 0x7ff9580b4e70 2024-03-25T19:22:37.995 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.994+0000 7ff94effd640 10 monclient(hunting): _negotiate 2024-03-25T19:22:37.995 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.994+0000 7ff94effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:37.995 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.994+0000 7ff94effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:37.995 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.994+0000 7ff94effd640 10 monclient(hunting): my global_id is 4605 2024-03-25T19:22:37.995 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:37.994+0000 7ff94effd640 1 -- 172.21.15.6:0/829124202 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9581438e0 con 0x7ff9580b4e70 2024-03-25T19:22:38.001 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.001+0000 7ff94effd640 1 -- 172.21.15.6:0/829124202 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 496473906 0 0) 0x7ff938004eb0 con 0x7ff9580b1b00 2024-03-25T19:22:38.001 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.001+0000 7ff94effd640 1 -- 172.21.15.6:0/829124202 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff958142d40 con 0x7ff9580b1b00 2024-03-25T19:22:38.001 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.001+0000 7ff94effd640 1 -- 172.21.15.6:0/829124202 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7ff948001a30 con 0x7ff9580b1b00 2024-03-25T19:22:38.001 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.001+0000 7ff94effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:38.002 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.001+0000 7ff94effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:22:38.002 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.001+0000 7ff94effd640 10 monclient(hunting): dump: 2024-03-25T19:22:38.002 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:22:38.002 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:38.002 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:38.002 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:38.002 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:38.002 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:38.002 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:22:38.002 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:22:38.002 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:22:38.002 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:38.002 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.001+0000 7ff94effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:22:38.006 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.007+0000 7ff94effd640 1 -- 172.21.15.6:0/829124202 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2541022403 0 0) 0x7ff958131920 con 0x7ff9580b44b0 2024-03-25T19:22:38.007 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.007+0000 7ff94effd640 10 monclient(hunting): _negotiate 2024-03-25T19:22:38.007 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.007+0000 7ff94effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:38.007 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.007+0000 7ff94effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:38.007 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.007+0000 7ff94effd640 10 monclient(hunting): my global_id is 4610 2024-03-25T19:22:38.007 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.007+0000 7ff94effd640 1 -- 172.21.15.6:0/829124202 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff938004eb0 con 0x7ff9580b44b0 2024-03-25T19:22:38.015 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.015+0000 7ff94effd640 1 -- 172.21.15.6:0/829124202 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2637067391 0 0) 0x7ff9581438e0 con 0x7ff9580b4e70 2024-03-25T19:22:38.015 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.015+0000 7ff94effd640 1 -- 172.21.15.6:0/829124202 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff958131920 con 0x7ff9580b4e70 2024-03-25T19:22:38.015 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.015+0000 7ff94effd640 1 -- 172.21.15.6:0/829124202 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7ff950001e80 con 0x7ff9580b4e70 2024-03-25T19:22:38.015 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.015+0000 7ff94effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:38.015 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.015+0000 7ff94effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:38.015 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.015+0000 7ff94effd640 10 monclient(hunting): dump: 2024-03-25T19:22:38.015 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:22:38.015 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:38.015 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:38.015 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:38.015 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:38.015 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:38.015 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:22:38.016 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:22:38.016 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:22:38.016 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:38.017 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.017+0000 7ff94effd640 1 -- 172.21.15.6:0/829124202 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 93341779 0 0) 0x7ff958142d40 con 0x7ff9580b1b00 2024-03-25T19:22:38.018 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.017+0000 7ff94effd640 1 -- 172.21.15.6:0/829124202 >> v1:172.21.15.6:6790/0 conn(0x7ff9580b44b0 legacy=0x7ff958130b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:38.018 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.017+0000 7ff94effd640 1 -- 172.21.15.6:0/829124202 >> v1:172.21.15.6:6789/0 conn(0x7ff9580b4e70 legacy=0x7ff958131210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:38.018 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.017+0000 7ff94effd640 10 monclient: _finish_hunting 0 2024-03-25T19:22:38.018 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.017+0000 7ff94effd640 1 monclient: found mon.b 2024-03-25T19:22:38.018 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.017+0000 7ff94effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:38.018 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.017+0000 7ff94effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:22:38.018 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.017+0000 7ff94effd640 1 -- 172.21.15.6:0/829124202 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff958144340 con 0x7ff9580b1b00 2024-03-25T19:22:38.018 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.017+0000 7ff94effd640 10 monclient: _finish_auth 0 2024-03-25T19:22:38.018 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.017+0000 7ff94effd640 10 monclient: _check_auth_tickets 2024-03-25T19:22:38.018 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.017+0000 7ff94effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:38.018 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.017+0000 7ff95cf97640 5 monclient: authenticate success, global_id 4618 2024-03-25T19:22:38.018 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.017+0000 7ff95cf97640 10 monclient: _renew_subs 2024-03-25T19:22:38.019 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.017+0000 7ff95cf97640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:22:38.019 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.017+0000 7ff95cf97640 1 -- 172.21.15.6:0/829124202 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff958144680 con 0x7ff9580b1b00 2024-03-25T19:22:38.019 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.017+0000 7ff95cf97640 10 client.4618.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:22:38.019 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.017+0000 7ff95cf97640 10 monclient: _renew_subs 2024-03-25T19:22:38.019 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.017+0000 7ff95cf97640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:22:38.019 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.017+0000 7ff95cf97640 1 -- 172.21.15.6:0/829124202 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff958142960 con 0x7ff9580b1b00 2024-03-25T19:22:38.019 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.017+0000 7ff95cf97640 1 librados: init done 2024-03-25T19:22:38.019 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.018+0000 7ff95cf97640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:38.019 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.018+0000 7ff95cf97640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:38.020 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.018+0000 7ff95cf97640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:22:38.020 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.018+0000 7ff95cf97640 1 -- 172.21.15.6:0/829124202 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff95804f730 con 0x7ff9580b1b00 2024-03-25T19:22:38.311 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.311+0000 7ff94effd640 1 -- 172.21.15.6:0/829124202 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff948001b90 con 0x7ff9580b1b00 2024-03-25T19:22:38.311 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.311+0000 7ff94effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:38.311 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.311+0000 7ff94effd640 1 -- 172.21.15.6:0/829124202 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7ff948003a60 con 0x7ff9580b1b00 2024-03-25T19:22:38.311 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.311+0000 7ff94effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:38.311 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.311+0000 7ff94effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:38.311 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.311+0000 7ff94effd640 10 monclient: dump: 2024-03-25T19:22:38.311 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:22:38.311 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:38.311 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:38.312 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:38.312 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:38.312 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:38.312 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:22:38.312 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:22:38.312 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:22:38.312 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:38.312 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.311+0000 7ff95559a640 4 set_mon_vals no callback set 2024-03-25T19:22:38.313 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.312+0000 7ff94effd640 1 -- 172.21.15.6:0/829124202 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7ff948003c80 con 0x7ff9580b1b00 2024-03-25T19:22:38.313 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.312+0000 7ff94ffff640 10 monclient: get_auth_request con 0x7ff93806dfb0 auth_method 0 2024-03-25T19:22:38.313 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.313+0000 7ff94effd640 10 client.4618.objecter ms_handle_connect 0x7ff93806dfb0 2024-03-25T19:22:38.317 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.317+0000 7ff94effd640 1 -- 172.21.15.6:0/829124202 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7ff9480547c0 con 0x7ff9580b1b00 2024-03-25T19:22:38.317 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.317+0000 7ff94effd640 10 client.4618.objecter ms_dispatch 0x7ff958000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:22:38.317 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.317+0000 7ff94effd640 3 client.4618.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:22:38.317 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.317+0000 7ff94effd640 3 client.4618.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:22:38.317 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.317+0000 7ff94effd640 20 client.4618.objecter dump_active .. 0 homeless 2024-03-25T19:22:38.317 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.317+0000 7ff94effd640 1 -- 172.21.15.6:0/829124202 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff9480502d0 con 0x7ff9580b1b00 2024-03-25T19:22:38.317 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.317+0000 7ff94effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:38.317 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.317+0000 7ff94effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:22:38.567 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.567+0000 7ff95cf97640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:38.567 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.567+0000 7ff95cf97640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:38.567 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.567+0000 7ff95cf97640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:22:38.567 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.567+0000 7ff95cf97640 1 -- 172.21.15.6:0/829124202 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9580aa090 con 0x7ff9580b1b00 2024-03-25T19:22:38.595 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.595+0000 7ff94effd640 1 -- 172.21.15.6:0/829124202 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7ff948052200 con 0x7ff9580b1b00 2024-03-25T19:22:38.595 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.595+0000 7ff94effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:38.595 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.595+0000 7ff94effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:22:38.596 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:22:38.596 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:22:38.598 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.597+0000 7ff95cf97640 10 librados: watch_flush enter 2024-03-25T19:22:38.598 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.597+0000 7ff95cf97640 10 librados: watch_flush exit 2024-03-25T19:22:38.598 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.597+0000 7ff95cf97640 20 client.4618.objecter shutdown clearing up homeless session... 2024-03-25T19:22:38.598 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.597+0000 7ff95cf97640 10 client.4618.objecter successfully canceled tick 2024-03-25T19:22:38.598 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.597+0000 7ff95cf97640 1 -- 172.21.15.6:0/829124202 >> v1:172.21.15.6:6800/2936652650 conn(0x7ff93806dfb0 legacy=0x7ff938070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:38.598 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.597+0000 7ff95cf97640 10 monclient: shutdown 2024-03-25T19:22:38.598 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.597+0000 7ff95cf97640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:38.598 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.597+0000 7ff95cf97640 1 -- 172.21.15.6:0/829124202 >> v1:172.21.15.192:6789/0 conn(0x7ff9580b1b00 legacy=0x7ff9581431d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:38.598 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.597+0000 7ff95cf97640 1 -- 172.21.15.6:0/829124202 shutdown_connections 2024-03-25T19:22:38.598 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.597+0000 7ff95cf97640 1 -- 172.21.15.6:0/829124202 >> 172.21.15.6:0/829124202 conn(0x7ff9580617a0 msgr2=0x7ff9580b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:38.598 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.597+0000 7ff95cf97640 1 -- 172.21.15.6:0/829124202 shutdown_connections 2024-03-25T19:22:38.599 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.597+0000 7ff95cf97640 1 -- 172.21.15.6:0/829124202 wait complete. 2024-03-25T19:22:38.599 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:38.597+0000 7ff95cf97640 1 librados: shutdown 2024-03-25T19:22:38.611 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:22:45.418 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:22:45.516 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.515+0000 7ff16b425640 10 monclient: get_monmap_and_config 2024-03-25T19:22:45.516 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.515+0000 7ff16b425640 10 monclient: build_initial_monmap 2024-03-25T19:22:45.516 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.515+0000 7ff16b425640 10 monclient: monmap: 2024-03-25T19:22:45.516 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:22:45.516 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:45.516 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:22:45.516825+0000 2024-03-25T19:22:45.516 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:22:45.516825+0000 2024-03-25T19:22:45.516 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:45.516 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:45.517 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:22:45.517 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:22:45.517 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:22:45.517 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:45.517 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.516+0000 7ff16b425640 1 Processor -- start 2024-03-25T19:22:45.517 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.516+0000 7ff16b425640 1 -- start start 2024-03-25T19:22:45.517 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.516+0000 7ff16b425640 10 monclient: init 2024-03-25T19:22:45.518 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.517+0000 7ff16b425640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:45.518 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.517+0000 7ff16b425640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:22:45.518 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.517+0000 7ff16b425640 10 monclient(hunting): picked mon.noname-c con 0x7ff1640b4e70 addr v1:172.21.15.192:6789/0 2024-03-25T19:22:45.518 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.517+0000 7ff16b425640 10 monclient(hunting): picked mon.noname-a con 0x7ff1640b44b0 addr v1:172.21.15.6:6789/0 2024-03-25T19:22:45.518 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.517+0000 7ff16b425640 10 monclient(hunting): picked mon.noname-b con 0x7ff1640b1b00 addr v1:172.21.15.6:6790/0 2024-03-25T19:22:45.519 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.517+0000 7ff16b425640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1640a96f0 con 0x7ff1640b44b0 2024-03-25T19:22:45.519 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.517+0000 7ff16b425640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1640ab0e0 con 0x7ff1640b4e70 2024-03-25T19:22:45.519 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.517+0000 7ff16b425640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff164145f70 con 0x7ff1640b1b00 2024-03-25T19:22:45.519 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.517+0000 7ff16b425640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:45.519 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.517+0000 7ff16b425640 10 monclient(hunting): authenticate will time out at 6469.757037s 2024-03-25T19:22:45.525 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.525+0000 7ff1689c0640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff1640b44b0 0x7ff164145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52018/0 (socket says 172.21.15.6:52018) 2024-03-25T19:22:45.525 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.525+0000 7ff1689c0640 1 -- 172.21.15.6:0/406331274 learned_addr learned my addr 172.21.15.6:0/406331274 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:22:45.535 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.535+0000 7ff15bfff640 1 -- 172.21.15.6:0/406331274 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 416329363 0 0) 0x7ff1640a96f0 con 0x7ff1640b44b0 2024-03-25T19:22:45.535 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.535+0000 7ff15bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:22:45.535 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.535+0000 7ff15bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:45.535 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.535+0000 7ff15bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:45.535 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.535+0000 7ff15bfff640 10 monclient(hunting): my global_id is 4608 2024-03-25T19:22:45.536 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.535+0000 7ff15bfff640 1 -- 172.21.15.6:0/406331274 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff150003520 con 0x7ff1640b44b0 2024-03-25T19:22:45.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.544+0000 7ff15bfff640 1 -- 172.21.15.6:0/406331274 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2895881546 0 0) 0x7ff150003520 con 0x7ff1640b44b0 2024-03-25T19:22:45.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.544+0000 7ff15bfff640 1 -- 172.21.15.6:0/406331274 >> v1:172.21.15.6:6790/0 conn(0x7ff1640b1b00 legacy=0x7ff16414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:22:45.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.544+0000 7ff15bfff640 1 -- 172.21.15.6:0/406331274 >> v1:172.21.15.192:6789/0 conn(0x7ff1640b4e70 legacy=0x7ff16413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:22:45.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.544+0000 7ff15bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:22:45.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.544+0000 7ff15bfff640 1 monclient: found mon.noname-a 2024-03-25T19:22:45.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.544+0000 7ff15bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:45.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.544+0000 7ff15bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-03-25T19:22:45.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.544+0000 7ff15bfff640 1 -- 172.21.15.6:0/406331274 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1640a9b80 con 0x7ff1640b44b0 2024-03-25T19:22:45.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.544+0000 7ff15bfff640 10 monclient: _finish_auth 0 2024-03-25T19:22:45.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.544+0000 7ff15bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:22:45.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.544+0000 7ff15bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:45.545 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.544+0000 7ff15bfff640 1 -- 172.21.15.6:0/406331274 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7ff154001690 con 0x7ff1640b44b0 2024-03-25T19:22:45.546 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.544+0000 7ff15bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:45.546 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.544+0000 7ff15bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:22:45.546 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.544+0000 7ff15bfff640 10 monclient: dump: 2024-03-25T19:22:45.546 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:22:45.546 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:45.546 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:45.546 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:45.546 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:45.546 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:45.546 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:22:45.546 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:22:45.546 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:22:45.546 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:45.546 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.544+0000 7ff16b425640 5 monclient: authenticate success, global_id 4608 2024-03-25T19:22:45.547 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.544+0000 7ff16b425640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:45.554 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.554+0000 7ff15bfff640 1 -- 172.21.15.6:0/406331274 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff154001860 con 0x7ff1640b44b0 2024-03-25T19:22:45.554 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.554+0000 7ff15bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:45.554 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.554+0000 7ff15bfff640 1 -- 172.21.15.6:0/406331274 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7ff154001cb0 con 0x7ff1640b44b0 2024-03-25T19:22:45.554 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.554+0000 7ff15bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:45.554 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.554+0000 7ff15bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:45.554 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.554+0000 7ff15bfff640 10 monclient: dump: 2024-03-25T19:22:45.555 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:22:45.555 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:45.555 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:45.555 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:45.555 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:45.555 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:45.555 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:22:45.555 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:22:45.555 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:22:45.555 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:45.555 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.554+0000 7ff16b425640 10 monclient: get_monmap_and_config success 2024-03-25T19:22:45.555 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.554+0000 7ff16b425640 4 set_mon_vals no callback set 2024-03-25T19:22:45.557 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.555+0000 7ff16b425640 10 monclient: shutdown 2024-03-25T19:22:45.557 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.555+0000 7ff16b425640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:45.557 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.555+0000 7ff16b425640 1 -- 172.21.15.6:0/406331274 >> v1:172.21.15.6:6789/0 conn(0x7ff1640b44b0 legacy=0x7ff164145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:45.557 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.555+0000 7ff16b425640 1 -- 172.21.15.6:0/406331274 shutdown_connections 2024-03-25T19:22:45.557 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.555+0000 7ff16b425640 1 -- 172.21.15.6:0/406331274 >> 172.21.15.6:0/406331274 conn(0x7ff1640617a0 msgr2=0x7ff164061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:45.557 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.555+0000 7ff16b425640 1 -- 172.21.15.6:0/406331274 shutdown_connections 2024-03-25T19:22:45.557 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.555+0000 7ff16b425640 1 -- 172.21.15.6:0/406331274 wait complete. 2024-03-25T19:22:45.557 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.556+0000 7ff16b425640 10 monclient: build_initial_monmap 2024-03-25T19:22:45.557 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.556+0000 7ff16b425640 10 monclient: monmap: 2024-03-25T19:22:45.557 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:22:45.558 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:22:45.558 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:22:45.558 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:22:45.558 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:45.558 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:22:45.558 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:22:45.558 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:22:45.558 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:22:45.558 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:45.558 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.557+0000 7ff16b425640 1 librados: starting msgr at 2024-03-25T19:22:45.558 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.557+0000 7ff16b425640 1 librados: starting objecter 2024-03-25T19:22:45.558 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.557+0000 7ff16b425640 1 Processor -- start 2024-03-25T19:22:45.558 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.557+0000 7ff16b425640 1 -- start start 2024-03-25T19:22:45.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.557+0000 7ff16b425640 1 librados: setting wanted keys 2024-03-25T19:22:45.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.557+0000 7ff16b425640 1 librados: calling monclient init 2024-03-25T19:22:45.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.557+0000 7ff16b425640 10 monclient: init 2024-03-25T19:22:45.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.558+0000 7ff16b425640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:45.560 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.558+0000 7ff16b425640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:22:45.560 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.558+0000 7ff16b425640 10 monclient(hunting): picked mon.noname-c con 0x7ff1640b1b00 addr v1:172.21.15.6:6790/0 2024-03-25T19:22:45.560 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.558+0000 7ff16b425640 10 monclient(hunting): picked mon.noname-b con 0x7ff1640b44b0 addr v1:172.21.15.192:6789/0 2024-03-25T19:22:45.560 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.558+0000 7ff16b425640 10 monclient(hunting): picked mon.noname-a con 0x7ff1640b4e70 addr v1:172.21.15.6:6789/0 2024-03-25T19:22:45.560 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.558+0000 7ff16b425640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff164142f90 con 0x7ff1640b4e70 2024-03-25T19:22:45.560 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.558+0000 7ff16b425640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff164141f10 con 0x7ff1640b44b0 2024-03-25T19:22:45.560 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.558+0000 7ff16b425640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1641431c0 con 0x7ff1640b1b00 2024-03-25T19:22:45.560 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.558+0000 7ff16b425640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:45.560 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.558+0000 7ff16b425640 10 monclient(hunting): authenticate will time out at 6469.798590s 2024-03-25T19:22:45.569 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.569+0000 7ff1689c0640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7ff1640b44b0 0x7ff164144a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:52780/0 (socket says 172.21.15.6:52780) 2024-03-25T19:22:45.569 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.569+0000 7ff1689c0640 1 -- 172.21.15.6:0/2976023950 learned_addr learned my addr 172.21.15.6:0/2976023950 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:22:45.572 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.572+0000 7ff159ffb640 10 client.?.objecter ms_handle_connect 0x7ff1640b44b0 2024-03-25T19:22:45.572 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.572+0000 7ff159ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:45.581 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.581+0000 7ff159ffb640 10 client.?.objecter ms_handle_connect 0x7ff1640b4e70 2024-03-25T19:22:45.581 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.581+0000 7ff159ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:45.588 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.588+0000 7ff159ffb640 1 -- 172.21.15.6:0/2976023950 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2090217282 0 0) 0x7ff164142f90 con 0x7ff1640b4e70 2024-03-25T19:22:45.588 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.588+0000 7ff159ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:22:45.588 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.588+0000 7ff159ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:45.588 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.588+0000 7ff159ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:45.589 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.588+0000 7ff159ffb640 10 monclient(hunting): my global_id is 4611 2024-03-25T19:22:45.589 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.588+0000 7ff159ffb640 1 -- 172.21.15.6:0/2976023950 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff148004eb0 con 0x7ff1640b4e70 2024-03-25T19:22:45.593 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.593+0000 7ff159ffb640 1 -- 172.21.15.6:0/2976023950 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 687354711 0 0) 0x7ff164141f10 con 0x7ff1640b44b0 2024-03-25T19:22:45.593 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.593+0000 7ff159ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:22:45.593 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.593+0000 7ff159ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:45.593 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.593+0000 7ff159ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:45.593 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.593+0000 7ff159ffb640 10 monclient(hunting): my global_id is 4621 2024-03-25T19:22:45.593 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.593+0000 7ff159ffb640 1 -- 172.21.15.6:0/2976023950 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff164142f90 con 0x7ff1640b44b0 2024-03-25T19:22:45.593 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.594+0000 7ff159ffb640 10 client.?.objecter ms_handle_connect 0x7ff1640b1b00 2024-03-25T19:22:45.593 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.594+0000 7ff159ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:45.604 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.604+0000 7ff159ffb640 1 -- 172.21.15.6:0/2976023950 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 171902252 0 0) 0x7ff1641431c0 con 0x7ff1640b1b00 2024-03-25T19:22:45.605 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.604+0000 7ff159ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:22:45.605 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.604+0000 7ff159ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:45.605 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.604+0000 7ff159ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:45.605 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.604+0000 7ff159ffb640 10 monclient(hunting): my global_id is 4613 2024-03-25T19:22:45.605 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.604+0000 7ff159ffb640 1 -- 172.21.15.6:0/2976023950 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff164141f10 con 0x7ff1640b1b00 2024-03-25T19:22:45.610 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.610+0000 7ff159ffb640 1 -- 172.21.15.6:0/2976023950 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3547570995 0 0) 0x7ff164142f90 con 0x7ff1640b44b0 2024-03-25T19:22:45.611 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.610+0000 7ff159ffb640 1 -- 172.21.15.6:0/2976023950 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1641431c0 con 0x7ff1640b44b0 2024-03-25T19:22:45.611 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.610+0000 7ff159ffb640 1 -- 172.21.15.6:0/2976023950 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7ff154001b70 con 0x7ff1640b44b0 2024-03-25T19:22:45.611 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.610+0000 7ff159ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:45.611 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.610+0000 7ff159ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:22:45.611 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.610+0000 7ff159ffb640 10 monclient(hunting): dump: 2024-03-25T19:22:45.611 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:22:45.611 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:45.611 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:45.611 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:45.611 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:45.611 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:45.611 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:22:45.611 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:22:45.612 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:22:45.612 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:45.612 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.610+0000 7ff159ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:22:45.614 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.614+0000 7ff159ffb640 1 -- 172.21.15.6:0/2976023950 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2061218445 0 0) 0x7ff148004eb0 con 0x7ff1640b4e70 2024-03-25T19:22:45.614 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.614+0000 7ff159ffb640 1 -- 172.21.15.6:0/2976023950 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff164142f90 con 0x7ff1640b4e70 2024-03-25T19:22:45.614 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.614+0000 7ff159ffb640 1 -- 172.21.15.6:0/2976023950 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7ff15c001b70 con 0x7ff1640b4e70 2024-03-25T19:22:45.614 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.614+0000 7ff159ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:45.614 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.614+0000 7ff159ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:45.614 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.614+0000 7ff159ffb640 10 monclient(hunting): dump: 2024-03-25T19:22:45.614 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:22:45.614 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:45.614 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:45.614 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:45.614 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:45.614 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:45.614 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:22:45.615 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:22:45.615 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:22:45.615 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:45.633 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.633+0000 7ff159ffb640 1 -- 172.21.15.6:0/2976023950 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3763511592 0 0) 0x7ff164141f10 con 0x7ff1640b1b00 2024-03-25T19:22:45.633 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.633+0000 7ff159ffb640 1 -- 172.21.15.6:0/2976023950 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff148004eb0 con 0x7ff1640b1b00 2024-03-25T19:22:45.634 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.633+0000 7ff159ffb640 1 -- 172.21.15.6:0/2976023950 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7ff1600016f0 con 0x7ff1640b1b00 2024-03-25T19:22:45.634 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.633+0000 7ff159ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:45.634 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.633+0000 7ff159ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:22:45.634 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.633+0000 7ff159ffb640 10 monclient(hunting): dump: 2024-03-25T19:22:45.634 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:22:45.634 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:45.634 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:45.634 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:45.634 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:45.634 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:45.634 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:22:45.634 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:22:45.635 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:22:45.635 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:45.647 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.646+0000 7ff159ffb640 1 -- 172.21.15.6:0/2976023950 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4003640577 0 0) 0x7ff1641431c0 con 0x7ff1640b44b0 2024-03-25T19:22:45.647 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.646+0000 7ff159ffb640 1 -- 172.21.15.6:0/2976023950 >> v1:172.21.15.6:6790/0 conn(0x7ff1640b1b00 legacy=0x7ff164140dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:45.647 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.647+0000 7ff159ffb640 1 -- 172.21.15.6:0/2976023950 >> v1:172.21.15.6:6789/0 conn(0x7ff1640b4e70 legacy=0x7ff164137e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:45.647 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.647+0000 7ff159ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:22:45.647 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.647+0000 7ff159ffb640 1 monclient: found mon.b 2024-03-25T19:22:45.647 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.647+0000 7ff159ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:45.647 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.647+0000 7ff159ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:22:45.648 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.647+0000 7ff159ffb640 1 -- 172.21.15.6:0/2976023950 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff164141ae0 con 0x7ff1640b44b0 2024-03-25T19:22:45.648 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.647+0000 7ff159ffb640 10 monclient: _finish_auth 0 2024-03-25T19:22:45.648 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.647+0000 7ff159ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:22:45.648 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.647+0000 7ff159ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:45.648 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.647+0000 7ff16b425640 5 monclient: authenticate success, global_id 4621 2024-03-25T19:22:45.648 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.647+0000 7ff16b425640 10 monclient: _renew_subs 2024-03-25T19:22:45.648 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.647+0000 7ff16b425640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:22:45.648 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.647+0000 7ff16b425640 1 -- 172.21.15.6:0/2976023950 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff164140160 con 0x7ff1640b44b0 2024-03-25T19:22:45.648 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.647+0000 7ff16b425640 10 client.4621.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:22:45.648 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.647+0000 7ff16b425640 10 monclient: _renew_subs 2024-03-25T19:22:45.648 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.647+0000 7ff16b425640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:22:45.648 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.647+0000 7ff16b425640 1 -- 172.21.15.6:0/2976023950 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff164145310 con 0x7ff1640b44b0 2024-03-25T19:22:45.648 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.647+0000 7ff16b425640 1 librados: init done 2024-03-25T19:22:45.649 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.648+0000 7ff16b425640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:45.649 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.648+0000 7ff16b425640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:45.649 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.648+0000 7ff16b425640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:22:45.649 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.648+0000 7ff16b425640 1 -- 172.21.15.6:0/2976023950 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff16404f730 con 0x7ff1640b44b0 2024-03-25T19:22:45.937 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.937+0000 7ff159ffb640 1 -- 172.21.15.6:0/2976023950 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff154001d70 con 0x7ff1640b44b0 2024-03-25T19:22:45.937 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.937+0000 7ff159ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:45.937 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.937+0000 7ff159ffb640 1 -- 172.21.15.6:0/2976023950 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7ff154004d90 con 0x7ff1640b44b0 2024-03-25T19:22:45.937 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.937+0000 7ff159ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:45.937 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.937+0000 7ff159ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:45.938 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.937+0000 7ff159ffb640 10 monclient: dump: 2024-03-25T19:22:45.938 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:22:45.938 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:45.938 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:45.938 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:45.938 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:45.938 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:45.938 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:22:45.938 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:22:45.938 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:22:45.938 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:45.938 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.937+0000 7ff15affd640 4 set_mon_vals no callback set 2024-03-25T19:22:45.939 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.938+0000 7ff159ffb640 1 -- 172.21.15.6:0/2976023950 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7ff15401ad60 con 0x7ff1640b44b0 2024-03-25T19:22:45.939 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.938+0000 7ff1691c1640 10 monclient: get_auth_request con 0x7ff14806e0b0 auth_method 0 2024-03-25T19:22:45.939 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.938+0000 7ff159ffb640 10 client.4621.objecter ms_handle_connect 0x7ff14806e0b0 2024-03-25T19:22:45.942 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.942+0000 7ff159ffb640 1 -- 172.21.15.6:0/2976023950 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7ff154054980 con 0x7ff1640b44b0 2024-03-25T19:22:45.942 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.942+0000 7ff159ffb640 10 client.4621.objecter ms_dispatch 0x7ff164000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:22:45.942 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.942+0000 7ff159ffb640 3 client.4621.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:22:45.943 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.942+0000 7ff159ffb640 3 client.4621.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:22:45.943 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.942+0000 7ff159ffb640 20 client.4621.objecter dump_active .. 0 homeless 2024-03-25T19:22:45.943 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.943+0000 7ff159ffb640 1 -- 172.21.15.6:0/2976023950 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff1540502c0 con 0x7ff1640b44b0 2024-03-25T19:22:45.943 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.943+0000 7ff159ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:45.943 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:45.943+0000 7ff159ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:22:46.202 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:46.202+0000 7ff16b425640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:46.202 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:46.202+0000 7ff16b425640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:46.202 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:46.202+0000 7ff16b425640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:22:46.202 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:46.202+0000 7ff16b425640 1 -- 172.21.15.6:0/2976023950 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1641420c0 con 0x7ff1640b44b0 2024-03-25T19:22:46.226 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:46.226+0000 7ff159ffb640 1 -- 172.21.15.6:0/2976023950 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7ff1641420c0 con 0x7ff1640b44b0 2024-03-25T19:22:46.226 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:46.226+0000 7ff159ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:46.226 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:46.226+0000 7ff159ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:22:46.227 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:22:46.227 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:22:46.229 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:46.229+0000 7ff16b425640 10 librados: watch_flush enter 2024-03-25T19:22:46.229 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:46.229+0000 7ff16b425640 10 librados: watch_flush exit 2024-03-25T19:22:46.229 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:46.229+0000 7ff16b425640 20 client.4621.objecter shutdown clearing up homeless session... 2024-03-25T19:22:46.230 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:46.229+0000 7ff16b425640 10 client.4621.objecter successfully canceled tick 2024-03-25T19:22:46.230 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:46.229+0000 7ff16b425640 1 -- 172.21.15.6:0/2976023950 >> v1:172.21.15.6:6800/2936652650 conn(0x7ff14806e0b0 legacy=0x7ff148070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:46.230 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:46.229+0000 7ff16b425640 10 monclient: shutdown 2024-03-25T19:22:46.230 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:46.229+0000 7ff16b425640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:46.230 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:46.229+0000 7ff16b425640 1 -- 172.21.15.6:0/2976023950 >> v1:172.21.15.192:6789/0 conn(0x7ff1640b44b0 legacy=0x7ff164144a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:46.230 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:46.229+0000 7ff16b425640 1 -- 172.21.15.6:0/2976023950 shutdown_connections 2024-03-25T19:22:46.230 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:46.229+0000 7ff16b425640 1 -- 172.21.15.6:0/2976023950 >> 172.21.15.6:0/2976023950 conn(0x7ff1640617a0 msgr2=0x7ff1640b59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:46.230 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:46.229+0000 7ff16b425640 1 -- 172.21.15.6:0/2976023950 shutdown_connections 2024-03-25T19:22:46.230 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:46.229+0000 7ff16b425640 1 -- 172.21.15.6:0/2976023950 wait complete. 2024-03-25T19:22:46.230 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:46.229+0000 7ff16b425640 1 librados: shutdown 2024-03-25T19:22:46.242 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:22:49.089 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:09:06.436 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:49.138 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:09:06.485 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:49.161 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:09:06.508 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:49.199 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:09:06.517 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:49.205 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:09:06.523 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:49.228 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:09:06.575 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:49.242 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:09:06.589 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:49.242 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:09:06.560 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:49.282 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:09:06.629 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:53.048 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:22:53.167 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.166+0000 7fb89bd96640 10 monclient: get_monmap_and_config 2024-03-25T19:22:53.167 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.166+0000 7fb89bd96640 10 monclient: build_initial_monmap 2024-03-25T19:22:53.167 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.166+0000 7fb89bd96640 10 monclient: monmap: 2024-03-25T19:22:53.167 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:22:53.167 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:53.167 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:22:53.168390+0000 2024-03-25T19:22:53.168 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:22:53.168390+0000 2024-03-25T19:22:53.168 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:53.168 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:53.168 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:22:53.168 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:22:53.168 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:22:53.168 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:53.169 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.166+0000 7fb89bd96640 1 Processor -- start 2024-03-25T19:22:53.169 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.167+0000 7fb89bd96640 1 -- start start 2024-03-25T19:22:53.169 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.167+0000 7fb89bd96640 10 monclient: init 2024-03-25T19:22:53.170 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.167+0000 7fb89bd96640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:53.170 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.167+0000 7fb89bd96640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:22:53.170 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.167+0000 7fb89bd96640 10 monclient(hunting): picked mon.noname-b con 0x7fb8940b4e70 addr v1:172.21.15.6:6790/0 2024-03-25T19:22:53.170 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.167+0000 7fb89bd96640 10 monclient(hunting): picked mon.noname-c con 0x7fb8940b44b0 addr v1:172.21.15.192:6789/0 2024-03-25T19:22:53.170 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.167+0000 7fb89bd96640 10 monclient(hunting): picked mon.noname-a con 0x7fb8940b1b00 addr v1:172.21.15.6:6789/0 2024-03-25T19:22:53.170 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.167+0000 7fb89bd96640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8940a96f0 con 0x7fb8940b1b00 2024-03-25T19:22:53.170 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.167+0000 7fb89bd96640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8940ab0e0 con 0x7fb8940b44b0 2024-03-25T19:22:53.170 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.167+0000 7fb89bd96640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb894145f70 con 0x7fb8940b4e70 2024-03-25T19:22:53.170 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.167+0000 7fb89bd96640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:53.170 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.167+0000 7fb89bd96640 10 monclient(hunting): authenticate will time out at 6477.408612s 2024-03-25T19:22:53.184 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.183+0000 7fb899b32640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fb8940b4e70 0x7fb89413f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:36878/0 (socket says 172.21.15.6:36878) 2024-03-25T19:22:53.184 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.183+0000 7fb899b32640 1 -- 172.21.15.6:0/3395379311 learned_addr learned my addr 172.21.15.6:0/3395379311 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:22:53.197 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.196+0000 7fb898b30640 1 -- 172.21.15.6:0/3395379311 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1002086983 0 0) 0x7fb8940ab0e0 con 0x7fb8940b44b0 2024-03-25T19:22:53.197 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.196+0000 7fb898b30640 10 monclient(hunting): _negotiate 2024-03-25T19:22:53.197 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.196+0000 7fb898b30640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:53.198 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.196+0000 7fb898b30640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:53.198 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.196+0000 7fb898b30640 10 monclient(hunting): my global_id is 4624 2024-03-25T19:22:53.198 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.197+0000 7fb898b30640 1 -- 172.21.15.6:0/3395379311 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb880003520 con 0x7fb8940b44b0 2024-03-25T19:22:53.204 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.203+0000 7fb898b30640 1 -- 172.21.15.6:0/3395379311 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1251647738 0 0) 0x7fb8940a96f0 con 0x7fb8940b1b00 2024-03-25T19:22:53.204 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.203+0000 7fb898b30640 10 monclient(hunting): _negotiate 2024-03-25T19:22:53.204 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.203+0000 7fb898b30640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:53.204 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.203+0000 7fb898b30640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:53.204 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.203+0000 7fb898b30640 10 monclient(hunting): my global_id is 4614 2024-03-25T19:22:53.204 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.203+0000 7fb898b30640 1 -- 172.21.15.6:0/3395379311 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8940ab0e0 con 0x7fb8940b1b00 2024-03-25T19:22:53.209 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.208+0000 7fb898b30640 1 -- 172.21.15.6:0/3395379311 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 395775648 0 0) 0x7fb894145f70 con 0x7fb8940b4e70 2024-03-25T19:22:53.209 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.208+0000 7fb898b30640 10 monclient(hunting): _negotiate 2024-03-25T19:22:53.209 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.208+0000 7fb898b30640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:53.209 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.208+0000 7fb898b30640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:53.209 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.208+0000 7fb898b30640 10 monclient(hunting): my global_id is 4616 2024-03-25T19:22:53.209 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.208+0000 7fb898b30640 1 -- 172.21.15.6:0/3395379311 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8940a96f0 con 0x7fb8940b4e70 2024-03-25T19:22:53.210 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.209+0000 7fb898b30640 1 -- 172.21.15.6:0/3395379311 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2786048304 0 0) 0x7fb8940ab0e0 con 0x7fb8940b1b00 2024-03-25T19:22:53.211 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.209+0000 7fb898b30640 1 -- 172.21.15.6:0/3395379311 >> v1:172.21.15.6:6790/0 conn(0x7fb8940b4e70 legacy=0x7fb89413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:53.211 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.209+0000 7fb898b30640 1 -- 172.21.15.6:0/3395379311 >> v1:172.21.15.192:6789/0 conn(0x7fb8940b44b0 legacy=0x7fb894145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:53.211 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.209+0000 7fb898b30640 10 monclient: _finish_hunting 0 2024-03-25T19:22:53.211 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.209+0000 7fb898b30640 1 monclient: found mon.noname-a 2024-03-25T19:22:53.211 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.209+0000 7fb898b30640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:53.211 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.209+0000 7fb898b30640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-03-25T19:22:53.211 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.209+0000 7fb898b30640 1 -- 172.21.15.6:0/3395379311 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8940a9b80 con 0x7fb8940b1b00 2024-03-25T19:22:53.211 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.209+0000 7fb898b30640 10 monclient: _finish_auth 0 2024-03-25T19:22:53.211 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.209+0000 7fb898b30640 10 monclient: _check_auth_tickets 2024-03-25T19:22:53.211 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.209+0000 7fb898b30640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:53.211 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.209+0000 7fb898b30640 1 -- 172.21.15.6:0/3395379311 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fb88c0019f0 con 0x7fb8940b1b00 2024-03-25T19:22:53.211 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.209+0000 7fb898b30640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:53.212 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.209+0000 7fb898b30640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:22:53.212 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.209+0000 7fb898b30640 10 monclient: dump: 2024-03-25T19:22:53.212 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:22:53.212 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:53.212 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:53.212 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:53.212 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:53.212 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:53.212 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:22:53.212 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:22:53.212 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:22:53.212 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:53.212 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.209+0000 7fb89bd96640 5 monclient: authenticate success, global_id 4614 2024-03-25T19:22:53.212 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.209+0000 7fb89bd96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:53.230 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.229+0000 7fb898b30640 1 -- 172.21.15.6:0/3395379311 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb88c001b50 con 0x7fb8940b1b00 2024-03-25T19:22:53.230 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.229+0000 7fb898b30640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:53.230 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.229+0000 7fb898b30640 1 -- 172.21.15.6:0/3395379311 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fb88c001f60 con 0x7fb8940b1b00 2024-03-25T19:22:53.230 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.229+0000 7fb898b30640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:53.230 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.229+0000 7fb898b30640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:53.230 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.229+0000 7fb898b30640 10 monclient: dump: 2024-03-25T19:22:53.231 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:22:53.231 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:53.231 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:53.231 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:53.231 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:53.231 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:53.231 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:22:53.231 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:22:53.231 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:22:53.231 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:53.231 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.229+0000 7fb89bd96640 10 monclient: get_monmap_and_config success 2024-03-25T19:22:53.231 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.229+0000 7fb89bd96640 4 set_mon_vals no callback set 2024-03-25T19:22:53.234 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.230+0000 7fb89bd96640 10 monclient: shutdown 2024-03-25T19:22:53.234 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.230+0000 7fb89bd96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:53.234 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.230+0000 7fb89bd96640 1 -- 172.21.15.6:0/3395379311 >> v1:172.21.15.6:6789/0 conn(0x7fb8940b1b00 legacy=0x7fb89414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:53.234 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.230+0000 7fb89bd96640 1 -- 172.21.15.6:0/3395379311 shutdown_connections 2024-03-25T19:22:53.234 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.230+0000 7fb89bd96640 1 -- 172.21.15.6:0/3395379311 >> 172.21.15.6:0/3395379311 conn(0x7fb8940617a0 msgr2=0x7fb894061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:53.234 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.230+0000 7fb89bd96640 1 -- 172.21.15.6:0/3395379311 shutdown_connections 2024-03-25T19:22:53.234 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.230+0000 7fb89bd96640 1 -- 172.21.15.6:0/3395379311 wait complete. 2024-03-25T19:22:53.234 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.231+0000 7fb89bd96640 10 monclient: build_initial_monmap 2024-03-25T19:22:53.235 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.231+0000 7fb89bd96640 10 monclient: monmap: 2024-03-25T19:22:53.235 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:22:53.235 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:22:53.235 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:22:53.235 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:22:53.235 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:53.235 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:22:53.235 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:22:53.235 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:22:53.235 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:22:53.235 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:53.235 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.231+0000 7fb89bd96640 1 librados: starting msgr at 2024-03-25T19:22:53.235 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.231+0000 7fb89bd96640 1 librados: starting objecter 2024-03-25T19:22:53.236 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.231+0000 7fb89bd96640 1 Processor -- start 2024-03-25T19:22:53.236 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.231+0000 7fb89bd96640 1 -- start start 2024-03-25T19:22:53.236 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.231+0000 7fb89bd96640 1 librados: setting wanted keys 2024-03-25T19:22:53.236 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.231+0000 7fb89bd96640 1 librados: calling monclient init 2024-03-25T19:22:53.236 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.231+0000 7fb89bd96640 10 monclient: init 2024-03-25T19:22:53.236 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.232+0000 7fb89bd96640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:53.236 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.232+0000 7fb89bd96640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:22:53.236 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.232+0000 7fb89bd96640 10 monclient(hunting): picked mon.noname-b con 0x7fb8940b1b00 addr v1:172.21.15.192:6789/0 2024-03-25T19:22:53.236 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.232+0000 7fb89bd96640 10 monclient(hunting): picked mon.noname-c con 0x7fb8940b44b0 addr v1:172.21.15.6:6790/0 2024-03-25T19:22:53.236 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.232+0000 7fb89bd96640 10 monclient(hunting): picked mon.noname-a con 0x7fb8940b4e70 addr v1:172.21.15.6:6789/0 2024-03-25T19:22:53.236 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.232+0000 7fb89bd96640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb894137460 con 0x7fb8940b4e70 2024-03-25T19:22:53.236 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.232+0000 7fb89bd96640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb894137740 con 0x7fb8940b1b00 2024-03-25T19:22:53.236 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.232+0000 7fb89bd96640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb894137a20 con 0x7fb8940b44b0 2024-03-25T19:22:53.236 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.232+0000 7fb89bd96640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:53.237 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.232+0000 7fb89bd96640 10 monclient(hunting): authenticate will time out at 6477.473256s 2024-03-25T19:22:53.242 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.241+0000 7fb89a333640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb8940b4e70 0x7fb8941388c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36266/0 (socket says 172.21.15.6:36266) 2024-03-25T19:22:53.242 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.241+0000 7fb89a333640 1 -- 172.21.15.6:0/3664577138 learned_addr learned my addr 172.21.15.6:0/3664577138 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:22:53.246 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.245+0000 7fb88a7fc640 10 client.?.objecter ms_handle_connect 0x7fb8940b4e70 2024-03-25T19:22:53.246 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.245+0000 7fb88a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:53.250 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.249+0000 7fb88a7fc640 10 client.?.objecter ms_handle_connect 0x7fb8940b44b0 2024-03-25T19:22:53.250 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.249+0000 7fb88a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:53.253 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.253+0000 7fb88a7fc640 10 client.?.objecter ms_handle_connect 0x7fb8940b1b00 2024-03-25T19:22:53.253 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.253+0000 7fb88a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:53.255 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.254+0000 7fb88a7fc640 1 -- 172.21.15.6:0/3664577138 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4114696813 0 0) 0x7fb894137a20 con 0x7fb8940b44b0 2024-03-25T19:22:53.255 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.254+0000 7fb88a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:22:53.255 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.254+0000 7fb88a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:53.255 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.254+0000 7fb88a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:53.255 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.254+0000 7fb88a7fc640 10 monclient(hunting): my global_id is 4619 2024-03-25T19:22:53.255 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.254+0000 7fb88a7fc640 1 -- 172.21.15.6:0/3664577138 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb878004eb0 con 0x7fb8940b44b0 2024-03-25T19:22:53.263 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.263+0000 7fb88a7fc640 1 -- 172.21.15.6:0/3664577138 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3465519224 0 0) 0x7fb894137460 con 0x7fb8940b4e70 2024-03-25T19:22:53.264 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.263+0000 7fb88a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:22:53.264 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.263+0000 7fb88a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:53.264 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.263+0000 7fb88a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:53.264 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.263+0000 7fb88a7fc640 10 monclient(hunting): my global_id is 4617 2024-03-25T19:22:53.264 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.263+0000 7fb88a7fc640 1 -- 172.21.15.6:0/3664577138 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb894137a20 con 0x7fb8940b4e70 2024-03-25T19:22:53.267 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.266+0000 7fb88a7fc640 1 -- 172.21.15.6:0/3664577138 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1701394385 0 0) 0x7fb894137740 con 0x7fb8940b1b00 2024-03-25T19:22:53.267 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.266+0000 7fb88a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:22:53.268 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.266+0000 7fb88a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:53.268 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.266+0000 7fb88a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:53.268 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.266+0000 7fb88a7fc640 10 monclient(hunting): my global_id is 4627 2024-03-25T19:22:53.268 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.266+0000 7fb88a7fc640 1 -- 172.21.15.6:0/3664577138 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb894137460 con 0x7fb8940b1b00 2024-03-25T19:22:53.273 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.272+0000 7fb88a7fc640 1 -- 172.21.15.6:0/3664577138 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 266021527 0 0) 0x7fb878004eb0 con 0x7fb8940b44b0 2024-03-25T19:22:53.273 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.272+0000 7fb88a7fc640 1 -- 172.21.15.6:0/3664577138 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb894137740 con 0x7fb8940b44b0 2024-03-25T19:22:53.273 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.272+0000 7fb88a7fc640 1 -- 172.21.15.6:0/3664577138 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fb884001950 con 0x7fb8940b44b0 2024-03-25T19:22:53.274 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.272+0000 7fb88a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:53.274 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.272+0000 7fb88a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:22:53.274 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.272+0000 7fb88a7fc640 10 monclient(hunting): dump: 2024-03-25T19:22:53.274 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:22:53.274 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:53.274 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:53.274 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:53.274 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:53.274 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:53.274 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:22:53.274 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:22:53.274 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:22:53.274 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:53.274 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.272+0000 7fb88a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:22:53.283 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.282+0000 7fb88a7fc640 1 -- 172.21.15.6:0/3664577138 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 13305910 0 0) 0x7fb894137a20 con 0x7fb8940b4e70 2024-03-25T19:22:53.283 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.282+0000 7fb88a7fc640 1 -- 172.21.15.6:0/3664577138 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb878004eb0 con 0x7fb8940b4e70 2024-03-25T19:22:53.283 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.282+0000 7fb88a7fc640 1 -- 172.21.15.6:0/3664577138 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fb88c001f20 con 0x7fb8940b4e70 2024-03-25T19:22:53.283 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.282+0000 7fb88a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:53.283 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.282+0000 7fb88a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:53.283 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.282+0000 7fb88a7fc640 10 monclient(hunting): dump: 2024-03-25T19:22:53.283 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:22:53.284 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:53.284 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:53.284 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:53.284 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:53.284 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:53.284 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:22:53.284 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:22:53.284 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:22:53.284 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:53.291 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.290+0000 7fb88a7fc640 1 -- 172.21.15.6:0/3664577138 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2062928003 0 0) 0x7fb894137740 con 0x7fb8940b44b0 2024-03-25T19:22:53.291 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.290+0000 7fb88a7fc640 1 -- 172.21.15.6:0/3664577138 >> v1:172.21.15.192:6789/0 conn(0x7fb8940b1b00 legacy=0x7fb894133e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:53.291 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.290+0000 7fb88a7fc640 1 -- 172.21.15.6:0/3664577138 >> v1:172.21.15.6:6789/0 conn(0x7fb8940b4e70 legacy=0x7fb8941388c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:53.292 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.290+0000 7fb88a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:22:53.292 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.290+0000 7fb88a7fc640 1 monclient: found mon.c 2024-03-25T19:22:53.292 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.290+0000 7fb88a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:53.292 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.290+0000 7fb88a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:22:53.292 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.290+0000 7fb88a7fc640 1 -- 172.21.15.6:0/3664577138 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb894136810 con 0x7fb8940b44b0 2024-03-25T19:22:53.292 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.290+0000 7fb88a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:22:53.292 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.290+0000 7fb88a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:22:53.292 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.290+0000 7fb88a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:53.292 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.290+0000 7fb89bd96640 5 monclient: authenticate success, global_id 4619 2024-03-25T19:22:53.292 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.290+0000 7fb89bd96640 10 monclient: _renew_subs 2024-03-25T19:22:53.292 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.290+0000 7fb89bd96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:22:53.292 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.290+0000 7fb89bd96640 1 -- 172.21.15.6:0/3664577138 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb894139520 con 0x7fb8940b44b0 2024-03-25T19:22:53.292 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.290+0000 7fb89bd96640 10 client.4619.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:22:53.292 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.290+0000 7fb89bd96640 10 monclient: _renew_subs 2024-03-25T19:22:53.292 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.290+0000 7fb89bd96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:22:53.293 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.290+0000 7fb89bd96640 1 -- 172.21.15.6:0/3664577138 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb894135af0 con 0x7fb8940b44b0 2024-03-25T19:22:53.293 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.290+0000 7fb89bd96640 1 librados: init done 2024-03-25T19:22:53.293 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.291+0000 7fb89bd96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:53.293 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.291+0000 7fb89bd96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:53.293 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.291+0000 7fb89bd96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:22:53.293 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.291+0000 7fb89bd96640 1 -- 172.21.15.6:0/3664577138 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb89404f730 con 0x7fb8940b44b0 2024-03-25T19:22:53.646 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.644+0000 7fb88a7fc640 1 -- 172.21.15.6:0/3664577138 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb884003610 con 0x7fb8940b44b0 2024-03-25T19:22:53.646 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.644+0000 7fb88a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:53.646 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.644+0000 7fb88a7fc640 1 -- 172.21.15.6:0/3664577138 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fb884001d80 con 0x7fb8940b44b0 2024-03-25T19:22:53.646 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.644+0000 7fb88a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:53.646 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.644+0000 7fb88a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:22:53.646 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.644+0000 7fb88a7fc640 10 monclient: dump: 2024-03-25T19:22:53.647 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:22:53.647 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:22:53.647 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:53.647 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:22:53.647 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:53.647 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:22:53.647 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:22:53.647 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:22:53.647 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:22:53.647 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:22:53.647 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.644+0000 7fb88bfff640 4 set_mon_vals no callback set 2024-03-25T19:22:53.648 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.645+0000 7fb88a7fc640 1 -- 172.21.15.6:0/3664577138 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7fb8840197d0 con 0x7fb8940b44b0 2024-03-25T19:22:53.648 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.646+0000 7fb899b32640 10 monclient: get_auth_request con 0x7fb87806df10 auth_method 0 2024-03-25T19:22:53.648 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.646+0000 7fb88a7fc640 10 client.4619.objecter ms_handle_connect 0x7fb87806df10 2024-03-25T19:22:53.652 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.651+0000 7fb88a7fc640 1 -- 172.21.15.6:0/3664577138 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7fb884053580 con 0x7fb8940b44b0 2024-03-25T19:22:53.652 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.651+0000 7fb88a7fc640 10 client.4619.objecter ms_dispatch 0x7fb894000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:22:53.652 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.651+0000 7fb88a7fc640 3 client.4619.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:22:53.652 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.651+0000 7fb88a7fc640 3 client.4619.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:22:53.652 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.651+0000 7fb88a7fc640 20 client.4619.objecter dump_active .. 0 homeless 2024-03-25T19:22:53.653 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.651+0000 7fb88a7fc640 1 -- 172.21.15.6:0/3664577138 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb88404eec0 con 0x7fb8940b44b0 2024-03-25T19:22:53.653 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.651+0000 7fb88a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:53.653 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.651+0000 7fb88a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:22:53.890 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.889+0000 7fb89bd96640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:53.890 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.889+0000 7fb89bd96640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:53.890 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.889+0000 7fb89bd96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:22:53.890 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.889+0000 7fb89bd96640 1 -- 172.21.15.6:0/3664577138 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8940aa090 con 0x7fb8940b44b0 2024-03-25T19:22:53.918 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.917+0000 7fb88a7fc640 1 -- 172.21.15.6:0/3664577138 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7fb884050fc0 con 0x7fb8940b44b0 2024-03-25T19:22:53.918 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.917+0000 7fb88a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:53.919 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.917+0000 7fb88a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:22:53.919 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:22:53.919 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:22:53.922 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.920+0000 7fb89bd96640 10 librados: watch_flush enter 2024-03-25T19:22:53.922 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.920+0000 7fb89bd96640 10 librados: watch_flush exit 2024-03-25T19:22:53.922 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.920+0000 7fb89bd96640 20 client.4619.objecter shutdown clearing up homeless session... 2024-03-25T19:22:53.922 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.920+0000 7fb89bd96640 10 client.4619.objecter successfully canceled tick 2024-03-25T19:22:53.922 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.920+0000 7fb89bd96640 1 -- 172.21.15.6:0/3664577138 >> v1:172.21.15.6:6800/2936652650 conn(0x7fb87806df10 legacy=0x7fb8780703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:53.922 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.920+0000 7fb89bd96640 10 monclient: shutdown 2024-03-25T19:22:53.922 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.920+0000 7fb89bd96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:53.922 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.920+0000 7fb89bd96640 1 -- 172.21.15.6:0/3664577138 >> v1:172.21.15.6:6790/0 conn(0x7fb8940b44b0 legacy=0x7fb8941381b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:53.922 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.920+0000 7fb89bd96640 1 -- 172.21.15.6:0/3664577138 shutdown_connections 2024-03-25T19:22:53.922 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.920+0000 7fb89bd96640 1 -- 172.21.15.6:0/3664577138 >> 172.21.15.6:0/3664577138 conn(0x7fb8940617a0 msgr2=0x7fb8940b6620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:53.923 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.920+0000 7fb89bd96640 1 -- 172.21.15.6:0/3664577138 shutdown_connections 2024-03-25T19:22:53.923 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.920+0000 7fb89bd96640 1 -- 172.21.15.6:0/3664577138 wait complete. 2024-03-25T19:22:53.923 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:22:53.920+0000 7fb89bd96640 1 librados: shutdown 2024-03-25T19:22:53.935 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:23:00.742 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:23:00.858 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.857+0000 7f28335d6640 10 monclient: get_monmap_and_config 2024-03-25T19:23:00.858 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.857+0000 7f28335d6640 10 monclient: build_initial_monmap 2024-03-25T19:23:00.858 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.857+0000 7f28335d6640 10 monclient: monmap: 2024-03-25T19:23:00.858 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:23:00.858 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:00.858 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:23:00.858962+0000 2024-03-25T19:23:00.858 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:23:00.858962+0000 2024-03-25T19:23:00.858 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:00.858 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:00.859 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:23:00.859 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:23:00.859 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:23:00.859 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:00.860 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.858+0000 7f28335d6640 1 Processor -- start 2024-03-25T19:23:00.860 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.859+0000 7f28335d6640 1 -- start start 2024-03-25T19:23:00.860 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.859+0000 7f28335d6640 10 monclient: init 2024-03-25T19:23:00.860 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.859+0000 7f28335d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:00.860 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.859+0000 7f28335d6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:23:00.860 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.859+0000 7f28335d6640 10 monclient(hunting): picked mon.noname-c con 0x7f282c0b4ff0 addr v1:172.21.15.192:6789/0 2024-03-25T19:23:00.860 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.859+0000 7f28335d6640 10 monclient(hunting): picked mon.noname-b con 0x7f282c0b4670 addr v1:172.21.15.6:6790/0 2024-03-25T19:23:00.861 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.859+0000 7f28335d6640 10 monclient(hunting): picked mon.noname-a con 0x7f282c13c480 addr v1:172.21.15.6:6789/0 2024-03-25T19:23:00.861 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.859+0000 7f28335d6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f282c0ab0e0 con 0x7f282c13c480 2024-03-25T19:23:00.861 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.859+0000 7f28335d6640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f282c0ab3c0 con 0x7f282c0b4ff0 2024-03-25T19:23:00.861 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.859+0000 7f28335d6640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f282c0a9f30 con 0x7f282c0b4670 2024-03-25T19:23:00.861 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.859+0000 7f28335d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:00.861 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.859+0000 7f28335d6640 10 monclient(hunting): authenticate will time out at 6485.099594s 2024-03-25T19:23:00.866 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.866+0000 7f2831b73640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f282c13c480 0x7f282c14dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33334/0 (socket says 172.21.15.6:33334) 2024-03-25T19:23:00.866 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.866+0000 7f2831b73640 1 -- 172.21.15.6:0/158499334 learned_addr learned my addr 172.21.15.6:0/158499334 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:23:00.874 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.874+0000 7f2823fff640 1 -- 172.21.15.6:0/158499334 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1045214972 0 0) 0x7f282c0ab0e0 con 0x7f282c13c480 2024-03-25T19:23:00.874 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.874+0000 7f2823fff640 10 monclient(hunting): _negotiate 2024-03-25T19:23:00.874 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.874+0000 7f2823fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:00.874 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.874+0000 7f2823fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:00.874 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.874+0000 7f2823fff640 10 monclient(hunting): my global_id is 4620 2024-03-25T19:23:00.874 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.874+0000 7f2823fff640 1 -- 172.21.15.6:0/158499334 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2818003520 con 0x7f282c13c480 2024-03-25T19:23:00.880 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.880+0000 7f2823fff640 1 -- 172.21.15.6:0/158499334 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4173512014 0 0) 0x7f282c0ab3c0 con 0x7f282c0b4ff0 2024-03-25T19:23:00.880 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.880+0000 7f2823fff640 10 monclient(hunting): _negotiate 2024-03-25T19:23:00.880 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.880+0000 7f2823fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:00.880 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.880+0000 7f2823fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:00.880 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.880+0000 7f2823fff640 10 monclient(hunting): my global_id is 4630 2024-03-25T19:23:00.880 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.880+0000 7f2823fff640 1 -- 172.21.15.6:0/158499334 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f282c0ab0e0 con 0x7f282c0b4ff0 2024-03-25T19:23:00.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.884+0000 7f2823fff640 1 -- 172.21.15.6:0/158499334 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3939491956 0 0) 0x7f2818003520 con 0x7f282c13c480 2024-03-25T19:23:00.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.884+0000 7f2823fff640 1 -- 172.21.15.6:0/158499334 >> v1:172.21.15.6:6790/0 conn(0x7f282c0b4670 legacy=0x7f282c1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:00.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.884+0000 7f2823fff640 1 -- 172.21.15.6:0/158499334 >> v1:172.21.15.192:6789/0 conn(0x7f282c0b4ff0 legacy=0x7f282c13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:00.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.884+0000 7f2823fff640 10 monclient: _finish_hunting 0 2024-03-25T19:23:00.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.884+0000 7f2823fff640 1 monclient: found mon.noname-a 2024-03-25T19:23:00.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.885+0000 7f2823fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:00.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.885+0000 7f2823fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-03-25T19:23:00.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.885+0000 7f2823fff640 1 -- 172.21.15.6:0/158499334 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f282c0a9b70 con 0x7f282c13c480 2024-03-25T19:23:00.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.885+0000 7f2823fff640 10 monclient: _finish_auth 0 2024-03-25T19:23:00.886 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.885+0000 7f2823fff640 10 monclient: _check_auth_tickets 2024-03-25T19:23:00.886 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.885+0000 7f2823fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:00.886 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.885+0000 7f2823fff640 1 -- 172.21.15.6:0/158499334 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f2824001a10 con 0x7f282c13c480 2024-03-25T19:23:00.886 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.885+0000 7f2823fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:00.886 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.885+0000 7f2823fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:23:00.886 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.885+0000 7f2823fff640 10 monclient: dump: 2024-03-25T19:23:00.886 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:23:00.886 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:00.886 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:00.886 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:00.886 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:00.886 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:00.886 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:23:00.887 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:23:00.887 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:23:00.887 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:00.887 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.885+0000 7f28335d6640 5 monclient: authenticate success, global_id 4620 2024-03-25T19:23:00.887 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.885+0000 7f28335d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:23:00.903 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.903+0000 7f2823fff640 1 -- 172.21.15.6:0/158499334 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2824001b70 con 0x7f282c13c480 2024-03-25T19:23:00.903 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.903+0000 7f2823fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:00.903 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.903+0000 7f2823fff640 1 -- 172.21.15.6:0/158499334 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f2824001f80 con 0x7f282c13c480 2024-03-25T19:23:00.903 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.903+0000 7f28335d6640 10 monclient: get_monmap_and_config success 2024-03-25T19:23:00.904 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.903+0000 7f28335d6640 4 set_mon_vals no callback set 2024-03-25T19:23:00.904 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.903+0000 7f2823fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:00.904 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.903+0000 7f2823fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:23:00.904 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.903+0000 7f2823fff640 10 monclient: dump: 2024-03-25T19:23:00.904 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:23:00.904 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:00.904 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:00.904 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:00.904 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:00.904 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:00.904 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:23:00.904 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:23:00.904 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:23:00.905 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:00.907 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.904+0000 7f28335d6640 10 monclient: shutdown 2024-03-25T19:23:00.907 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.904+0000 7f28335d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:00.907 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.904+0000 7f28335d6640 1 -- 172.21.15.6:0/158499334 >> v1:172.21.15.6:6789/0 conn(0x7f282c13c480 legacy=0x7f282c14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:00.907 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.904+0000 7f28335d6640 1 -- 172.21.15.6:0/158499334 shutdown_connections 2024-03-25T19:23:00.907 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.904+0000 7f28335d6640 1 -- 172.21.15.6:0/158499334 >> 172.21.15.6:0/158499334 conn(0x7f282c0617a0 msgr2=0x7f282c061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:00.907 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.904+0000 7f28335d6640 1 -- 172.21.15.6:0/158499334 shutdown_connections 2024-03-25T19:23:00.907 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.904+0000 7f28335d6640 1 -- 172.21.15.6:0/158499334 wait complete. 2024-03-25T19:23:00.907 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.905+0000 7f28335d6640 10 monclient: build_initial_monmap 2024-03-25T19:23:00.907 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.905+0000 7f28335d6640 10 monclient: monmap: 2024-03-25T19:23:00.907 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:23:00.908 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:23:00.908 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:23:00.908 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:23:00.908 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:00.908 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:23:00.908 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:23:00.908 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:23:00.908 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:23:00.908 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:00.908 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.905+0000 7f28335d6640 1 librados: starting msgr at 2024-03-25T19:23:00.908 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.905+0000 7f28335d6640 1 librados: starting objecter 2024-03-25T19:23:00.908 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.906+0000 7f28335d6640 1 Processor -- start 2024-03-25T19:23:00.909 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.906+0000 7f28335d6640 1 -- start start 2024-03-25T19:23:00.909 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.906+0000 7f28335d6640 1 librados: setting wanted keys 2024-03-25T19:23:00.909 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.906+0000 7f28335d6640 1 librados: calling monclient init 2024-03-25T19:23:00.909 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.906+0000 7f28335d6640 10 monclient: init 2024-03-25T19:23:00.909 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.906+0000 7f28335d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:00.909 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.906+0000 7f28335d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:23:00.909 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.906+0000 7f28335d6640 10 monclient(hunting): picked mon.noname-b con 0x7f282c0b4670 addr v1:172.21.15.192:6789/0 2024-03-25T19:23:00.909 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.906+0000 7f28335d6640 10 monclient(hunting): picked mon.noname-a con 0x7f282c0b4ff0 addr v1:172.21.15.6:6789/0 2024-03-25T19:23:00.909 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.906+0000 7f28335d6640 10 monclient(hunting): picked mon.noname-c con 0x7f282c13c480 addr v1:172.21.15.6:6790/0 2024-03-25T19:23:00.910 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.906+0000 7f28335d6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f282c0b0160 con 0x7f282c0b4ff0 2024-03-25T19:23:00.910 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.906+0000 7f28335d6640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f282c0b0440 con 0x7f282c0b4670 2024-03-25T19:23:00.910 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.906+0000 7f28335d6640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f282c0b0720 con 0x7f282c13c480 2024-03-25T19:23:00.910 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.906+0000 7f28335d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:00.910 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.906+0000 7f28335d6640 10 monclient(hunting): authenticate will time out at 6485.146520s 2024-03-25T19:23:00.910 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.909+0000 7f2831b73640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f282c13c480 0x7f282c0aeb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:33386/0 (socket says 172.21.15.6:33386) 2024-03-25T19:23:00.911 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.909+0000 7f2831b73640 1 -- 172.21.15.6:0/1195593431 learned_addr learned my addr 172.21.15.6:0/1195593431 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:23:00.911 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.910+0000 7f2821ffb640 10 client.?.objecter ms_handle_connect 0x7f282c13c480 2024-03-25T19:23:00.911 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.910+0000 7f2821ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:00.919 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.920+0000 7f2821ffb640 10 client.?.objecter ms_handle_connect 0x7f282c0b4ff0 2024-03-25T19:23:00.919 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.920+0000 7f2821ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:00.921 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.921+0000 7f2821ffb640 10 client.?.objecter ms_handle_connect 0x7f282c0b4670 2024-03-25T19:23:00.921 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.921+0000 7f2821ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:00.922 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.922+0000 7f2821ffb640 1 -- 172.21.15.6:0/1195593431 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 923518152 0 0) 0x7f282c0b0720 con 0x7f282c13c480 2024-03-25T19:23:00.922 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.922+0000 7f2821ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:23:00.922 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.922+0000 7f2821ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:00.922 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.922+0000 7f2821ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:00.922 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.922+0000 7f2821ffb640 10 monclient(hunting): my global_id is 4625 2024-03-25T19:23:00.922 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.922+0000 7f2821ffb640 1 -- 172.21.15.6:0/1195593431 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2810004eb0 con 0x7f282c13c480 2024-03-25T19:23:00.928 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.928+0000 7f2821ffb640 1 -- 172.21.15.6:0/1195593431 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2717657756 0 0) 0x7f282c0b0160 con 0x7f282c0b4ff0 2024-03-25T19:23:00.929 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.928+0000 7f2821ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:23:00.929 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.928+0000 7f2821ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:00.929 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.928+0000 7f2821ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:00.929 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.928+0000 7f2821ffb640 10 monclient(hunting): my global_id is 4623 2024-03-25T19:23:00.929 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.928+0000 7f2821ffb640 1 -- 172.21.15.6:0/1195593431 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f282c0b0720 con 0x7f282c0b4ff0 2024-03-25T19:23:00.936 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.936+0000 7f2821ffb640 1 -- 172.21.15.6:0/1195593431 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1578641742 0 0) 0x7f282c0b0440 con 0x7f282c0b4670 2024-03-25T19:23:00.936 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.936+0000 7f2821ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:23:00.936 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.936+0000 7f2821ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:00.936 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.936+0000 7f2821ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:00.936 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.936+0000 7f2821ffb640 10 monclient(hunting): my global_id is 4633 2024-03-25T19:23:00.936 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.936+0000 7f2821ffb640 1 -- 172.21.15.6:0/1195593431 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f282c0b0160 con 0x7f282c0b4670 2024-03-25T19:23:00.939 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.939+0000 7f2821ffb640 1 -- 172.21.15.6:0/1195593431 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2906977631 0 0) 0x7f282c0b0720 con 0x7f282c0b4ff0 2024-03-25T19:23:00.939 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.939+0000 7f2821ffb640 1 -- 172.21.15.6:0/1195593431 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f282c0b0440 con 0x7f282c0b4ff0 2024-03-25T19:23:00.939 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.939+0000 7f2821ffb640 1 -- 172.21.15.6:0/1195593431 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f281c0018a0 con 0x7f282c0b4ff0 2024-03-25T19:23:00.939 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.939+0000 7f2821ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:00.940 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.939+0000 7f2821ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:23:00.940 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.939+0000 7f2821ffb640 10 monclient(hunting): dump: 2024-03-25T19:23:00.940 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:23:00.940 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:00.940 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:00.940 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:00.940 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:00.940 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:00.940 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:23:00.940 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:23:00.940 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:23:00.940 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:00.940 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.939+0000 7f2821ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:23:00.943 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.943+0000 7f2821ffb640 1 -- 172.21.15.6:0/1195593431 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3172454787 0 0) 0x7f2810004eb0 con 0x7f282c13c480 2024-03-25T19:23:00.943 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.943+0000 7f2821ffb640 1 -- 172.21.15.6:0/1195593431 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f282c0b0720 con 0x7f282c13c480 2024-03-25T19:23:00.943 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.943+0000 7f2821ffb640 1 -- 172.21.15.6:0/1195593431 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f2824001ec0 con 0x7f282c13c480 2024-03-25T19:23:00.943 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.943+0000 7f2821ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:00.943 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.943+0000 7f2821ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:23:00.943 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.943+0000 7f2821ffb640 10 monclient(hunting): dump: 2024-03-25T19:23:00.943 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:23:00.943 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:00.943 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:00.943 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:00.943 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:00.944 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:00.944 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:23:00.944 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:23:00.944 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:23:00.944 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:00.952 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.952+0000 7f2821ffb640 1 -- 172.21.15.6:0/1195593431 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 760217501 0 0) 0x7f282c0b0720 con 0x7f282c13c480 2024-03-25T19:23:00.952 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.952+0000 7f2821ffb640 1 -- 172.21.15.6:0/1195593431 >> v1:172.21.15.192:6789/0 conn(0x7f282c0b4670 legacy=0x7f282c0b0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:00.952 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.952+0000 7f2821ffb640 1 -- 172.21.15.6:0/1195593431 >> v1:172.21.15.6:6789/0 conn(0x7f282c0b4ff0 legacy=0x7f282c0ae430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:00.952 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.952+0000 7f2821ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:23:00.953 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.952+0000 7f2821ffb640 1 monclient: found mon.c 2024-03-25T19:23:00.953 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.952+0000 7f2821ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:00.953 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.952+0000 7f2821ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:23:00.953 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.952+0000 7f2821ffb640 1 -- 172.21.15.6:0/1195593431 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f282c0b15c0 con 0x7f282c13c480 2024-03-25T19:23:00.953 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.952+0000 7f2821ffb640 10 monclient: _finish_auth 0 2024-03-25T19:23:00.953 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.952+0000 7f2821ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:23:00.953 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.952+0000 7f2821ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:00.953 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.952+0000 7f28335d6640 5 monclient: authenticate success, global_id 4625 2024-03-25T19:23:00.953 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.952+0000 7f28335d6640 10 monclient: _renew_subs 2024-03-25T19:23:00.953 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.952+0000 7f28335d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:23:00.953 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.952+0000 7f28335d6640 1 -- 172.21.15.6:0/1195593431 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f282c1e35a0 con 0x7f282c13c480 2024-03-25T19:23:00.953 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.952+0000 7f28335d6640 10 client.4625.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:23:00.953 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.952+0000 7f28335d6640 10 monclient: _renew_subs 2024-03-25T19:23:00.953 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.952+0000 7f28335d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:23:00.953 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.952+0000 7f28335d6640 1 -- 172.21.15.6:0/1195593431 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f282c1e3910 con 0x7f282c13c480 2024-03-25T19:23:00.953 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.952+0000 7f28335d6640 1 librados: init done 2024-03-25T19:23:00.954 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.953+0000 7f28335d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:00.954 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.953+0000 7f28335d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:00.954 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.953+0000 7f28335d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:23:00.954 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:00.953+0000 7f28335d6640 1 -- 172.21.15.6:0/1195593431 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f282c0ae150 con 0x7f282c13c480 2024-03-25T19:23:01.274 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:01.274+0000 7f2821ffb640 1 -- 172.21.15.6:0/1195593431 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2824004e10 con 0x7f282c13c480 2024-03-25T19:23:01.274 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:01.274+0000 7f2821ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:01.274 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:01.274+0000 7f2821ffb640 1 -- 172.21.15.6:0/1195593431 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f2824002100 con 0x7f282c13c480 2024-03-25T19:23:01.275 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:01.274+0000 7f2821ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:01.275 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:01.274+0000 7f2821ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:23:01.275 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:01.274+0000 7f2821ffb640 10 monclient: dump: 2024-03-25T19:23:01.275 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:23:01.275 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:01.275 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:01.275 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:01.275 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:01.275 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:01.275 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:23:01.275 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:23:01.275 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:23:01.275 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:01.275 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:01.274+0000 7f28237fe640 4 set_mon_vals no callback set 2024-03-25T19:23:01.276 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:01.275+0000 7f2821ffb640 1 -- 172.21.15.6:0/1195593431 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f282401adf0 con 0x7f282c13c480 2024-03-25T19:23:01.276 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:01.275+0000 7f2831372640 10 monclient: get_auth_request con 0x7f281006dfb0 auth_method 0 2024-03-25T19:23:01.276 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:01.275+0000 7f2821ffb640 10 client.4625.objecter ms_handle_connect 0x7f281006dfb0 2024-03-25T19:23:01.281 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:01.281+0000 7f2821ffb640 1 -- 172.21.15.6:0/1195593431 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f28240549f0 con 0x7f282c13c480 2024-03-25T19:23:01.281 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:01.281+0000 7f2821ffb640 10 client.4625.objecter ms_dispatch 0x7f282c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:23:01.281 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:01.281+0000 7f2821ffb640 3 client.4625.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:23:01.281 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:01.281+0000 7f2821ffb640 3 client.4625.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:23:01.281 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:01.281+0000 7f2821ffb640 20 client.4625.objecter dump_active .. 0 homeless 2024-03-25T19:23:01.281 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:01.281+0000 7f2821ffb640 1 -- 172.21.15.6:0/1195593431 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2824050330 con 0x7f282c13c480 2024-03-25T19:23:01.281 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:01.281+0000 7f2821ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:01.282 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:01.281+0000 7f2821ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:23:01.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:01.506+0000 7f28335d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:01.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:01.506+0000 7f28335d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:01.507 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:01.506+0000 7f28335d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:23:01.507 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:01.506+0000 7f28335d6640 1 -- 172.21.15.6:0/1195593431 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f282c06dbd0 con 0x7f282c13c480 2024-03-25T19:23:01.530 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:01.530+0000 7f2821ffb640 1 -- 172.21.15.6:0/1195593431 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f2824052430 con 0x7f282c13c480 2024-03-25T19:23:01.531 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:01.530+0000 7f2821ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:01.531 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:01.530+0000 7f2821ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:23:01.531 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:23:01.531 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:23:01.533 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:01.533+0000 7f28335d6640 10 librados: watch_flush enter 2024-03-25T19:23:01.533 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:01.533+0000 7f28335d6640 10 librados: watch_flush exit 2024-03-25T19:23:01.533 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:01.533+0000 7f28335d6640 20 client.4625.objecter shutdown clearing up homeless session... 2024-03-25T19:23:01.533 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:01.533+0000 7f28335d6640 10 client.4625.objecter successfully canceled tick 2024-03-25T19:23:01.533 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:01.533+0000 7f28335d6640 1 -- 172.21.15.6:0/1195593431 >> v1:172.21.15.6:6800/2936652650 conn(0x7f281006dfb0 legacy=0x7f2810070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:01.533 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:01.533+0000 7f28335d6640 10 monclient: shutdown 2024-03-25T19:23:01.533 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:01.533+0000 7f28335d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:01.534 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:01.533+0000 7f28335d6640 1 -- 172.21.15.6:0/1195593431 >> v1:172.21.15.6:6790/0 conn(0x7f282c13c480 legacy=0x7f282c0aeb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:01.534 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:01.533+0000 7f28335d6640 1 -- 172.21.15.6:0/1195593431 shutdown_connections 2024-03-25T19:23:01.534 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:01.533+0000 7f28335d6640 1 -- 172.21.15.6:0/1195593431 >> 172.21.15.6:0/1195593431 conn(0x7f282c0617a0 msgr2=0x7f282c145420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:01.534 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:01.533+0000 7f28335d6640 1 -- 172.21.15.6:0/1195593431 shutdown_connections 2024-03-25T19:23:01.534 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:01.533+0000 7f28335d6640 1 -- 172.21.15.6:0/1195593431 wait complete. 2024-03-25T19:23:01.534 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:01.533+0000 7f28335d6640 1 librados: shutdown 2024-03-25T19:23:01.545 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:23:08.415 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:23:08.530 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.530+0000 7febacb04640 10 monclient: get_monmap_and_config 2024-03-25T19:23:08.530 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.530+0000 7febacb04640 10 monclient: build_initial_monmap 2024-03-25T19:23:08.530 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.530+0000 7febacb04640 10 monclient: monmap: 2024-03-25T19:23:08.530 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:23:08.530 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:08.531 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:23:08.531720+0000 2024-03-25T19:23:08.531 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:23:08.531720+0000 2024-03-25T19:23:08.531 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:08.531 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:08.531 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:23:08.531 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:23:08.531 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:23:08.531 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:08.532 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.531+0000 7febacb04640 1 Processor -- start 2024-03-25T19:23:08.532 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.531+0000 7febacb04640 1 -- start start 2024-03-25T19:23:08.532 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.531+0000 7febacb04640 10 monclient: init 2024-03-25T19:23:08.533 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.532+0000 7febacb04640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:08.533 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.532+0000 7febacb04640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:23:08.533 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.532+0000 7febacb04640 10 monclient(hunting): picked mon.noname-c con 0x7feba80b4ff0 addr v1:172.21.15.192:6789/0 2024-03-25T19:23:08.533 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.532+0000 7febacb04640 10 monclient(hunting): picked mon.noname-a con 0x7feba80b4670 addr v1:172.21.15.6:6789/0 2024-03-25T19:23:08.533 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.532+0000 7febacb04640 10 monclient(hunting): picked mon.noname-b con 0x7feba813c480 addr v1:172.21.15.6:6790/0 2024-03-25T19:23:08.533 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.532+0000 7febacb04640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba80ab0e0 con 0x7feba80b4670 2024-03-25T19:23:08.533 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.532+0000 7febacb04640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba80ab3c0 con 0x7feba80b4ff0 2024-03-25T19:23:08.533 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.532+0000 7febacb04640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba80a9f30 con 0x7feba813c480 2024-03-25T19:23:08.533 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.532+0000 7febacb04640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:08.533 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.532+0000 7febacb04640 10 monclient(hunting): authenticate will time out at 6492.772319s 2024-03-25T19:23:08.544 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.544+0000 7feba659c640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7feba80b4ff0 0x7feba813bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:59482/0 (socket says 172.21.15.6:59482) 2024-03-25T19:23:08.544 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.544+0000 7feba659c640 1 -- 172.21.15.6:0/1192716390 learned_addr learned my addr 172.21.15.6:0/1192716390 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:23:08.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.552+0000 7feba559a640 1 -- 172.21.15.6:0/1192716390 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2653216791 0 0) 0x7feba80ab3c0 con 0x7feba80b4ff0 2024-03-25T19:23:08.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.552+0000 7feba559a640 10 monclient(hunting): _negotiate 2024-03-25T19:23:08.553 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.552+0000 7feba559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:08.553 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.552+0000 7feba559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:08.553 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.552+0000 7feba559a640 10 monclient(hunting): my global_id is 4636 2024-03-25T19:23:08.553 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.552+0000 7feba559a640 1 -- 172.21.15.6:0/1192716390 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb94003520 con 0x7feba80b4ff0 2024-03-25T19:23:08.555 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.555+0000 7feba559a640 1 -- 172.21.15.6:0/1192716390 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2256321834 0 0) 0x7feba80a9f30 con 0x7feba813c480 2024-03-25T19:23:08.555 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.555+0000 7feba559a640 10 monclient(hunting): _negotiate 2024-03-25T19:23:08.555 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.555+0000 7feba559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:08.556 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.555+0000 7feba559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:08.556 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.555+0000 7feba559a640 10 monclient(hunting): my global_id is 4628 2024-03-25T19:23:08.556 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.555+0000 7feba559a640 1 -- 172.21.15.6:0/1192716390 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb94003840 con 0x7feba813c480 2024-03-25T19:23:08.569 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.568+0000 7feba559a640 1 -- 172.21.15.6:0/1192716390 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2435992602 0 0) 0x7feb94003520 con 0x7feba80b4ff0 2024-03-25T19:23:08.569 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.568+0000 7feba559a640 1 -- 172.21.15.6:0/1192716390 >> v1:172.21.15.6:6790/0 conn(0x7feba813c480 legacy=0x7feba814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:08.569 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.568+0000 7feba559a640 1 -- 172.21.15.6:0/1192716390 >> v1:172.21.15.6:6789/0 conn(0x7feba80b4670 legacy=0x7feba81425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:08.569 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.568+0000 7feba559a640 10 monclient: _finish_hunting 0 2024-03-25T19:23:08.569 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.568+0000 7feba559a640 1 monclient: found mon.noname-c 2024-03-25T19:23:08.569 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.568+0000 7feba559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:08.569 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.568+0000 7feba559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:23:08.569 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.568+0000 7feba559a640 1 -- 172.21.15.6:0/1192716390 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba80a9b70 con 0x7feba80b4ff0 2024-03-25T19:23:08.569 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.568+0000 7feba559a640 10 monclient: _finish_auth 0 2024-03-25T19:23:08.569 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.568+0000 7feba559a640 10 monclient: _check_auth_tickets 2024-03-25T19:23:08.569 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.568+0000 7feba559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:08.569 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.568+0000 7feba559a640 1 -- 172.21.15.6:0/1192716390 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7feba00019f0 con 0x7feba80b4ff0 2024-03-25T19:23:08.570 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.569+0000 7feba559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:08.570 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.569+0000 7feba559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:23:08.570 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.569+0000 7feba559a640 10 monclient: dump: 2024-03-25T19:23:08.570 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:23:08.570 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:08.570 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:08.570 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:08.570 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:08.570 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:08.570 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:23:08.570 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:23:08.570 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:23:08.570 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:08.570 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.569+0000 7febacb04640 5 monclient: authenticate success, global_id 4636 2024-03-25T19:23:08.571 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.569+0000 7febacb04640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:23:08.590 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.590+0000 7feba559a640 1 -- 172.21.15.6:0/1192716390 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feba0001b50 con 0x7feba80b4ff0 2024-03-25T19:23:08.590 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.590+0000 7feba559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:08.590 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.590+0000 7feba559a640 1 -- 172.21.15.6:0/1192716390 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7feba0001f60 con 0x7feba80b4ff0 2024-03-25T19:23:08.590 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.590+0000 7feba559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:08.590 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.590+0000 7feba559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:23:08.590 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.590+0000 7feba559a640 10 monclient: dump: 2024-03-25T19:23:08.590 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:23:08.590 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:08.590 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:08.591 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:08.591 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:08.591 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:08.591 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:23:08.591 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:23:08.591 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:23:08.591 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:08.591 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.590+0000 7febacb04640 10 monclient: get_monmap_and_config success 2024-03-25T19:23:08.591 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.590+0000 7febacb04640 4 set_mon_vals no callback set 2024-03-25T19:23:08.592 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.591+0000 7febacb04640 10 monclient: shutdown 2024-03-25T19:23:08.592 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.591+0000 7febacb04640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:08.592 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.591+0000 7febacb04640 1 -- 172.21.15.6:0/1192716390 >> v1:172.21.15.192:6789/0 conn(0x7feba80b4ff0 legacy=0x7feba813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:08.592 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.591+0000 7febacb04640 1 -- 172.21.15.6:0/1192716390 shutdown_connections 2024-03-25T19:23:08.592 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.591+0000 7febacb04640 1 -- 172.21.15.6:0/1192716390 >> 172.21.15.6:0/1192716390 conn(0x7feba80617a0 msgr2=0x7feba8061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:08.592 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.591+0000 7febacb04640 1 -- 172.21.15.6:0/1192716390 shutdown_connections 2024-03-25T19:23:08.592 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.591+0000 7febacb04640 1 -- 172.21.15.6:0/1192716390 wait complete. 2024-03-25T19:23:08.593 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.592+0000 7febacb04640 10 monclient: build_initial_monmap 2024-03-25T19:23:08.593 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.592+0000 7febacb04640 10 monclient: monmap: 2024-03-25T19:23:08.593 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:23:08.593 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:23:08.593 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:23:08.593 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:23:08.593 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:08.593 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:23:08.593 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:23:08.593 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:23:08.593 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:23:08.593 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:08.595 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.593+0000 7febacb04640 1 librados: starting msgr at 2024-03-25T19:23:08.595 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.593+0000 7febacb04640 1 librados: starting objecter 2024-03-25T19:23:08.595 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.593+0000 7febacb04640 1 Processor -- start 2024-03-25T19:23:08.595 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.593+0000 7febacb04640 1 -- start start 2024-03-25T19:23:08.595 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.593+0000 7febacb04640 1 librados: setting wanted keys 2024-03-25T19:23:08.595 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.593+0000 7febacb04640 1 librados: calling monclient init 2024-03-25T19:23:08.595 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.593+0000 7febacb04640 10 monclient: init 2024-03-25T19:23:08.595 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.593+0000 7febacb04640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:08.595 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.593+0000 7febacb04640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:23:08.595 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.594+0000 7febacb04640 10 monclient(hunting): picked mon.noname-a con 0x7feba80b4670 addr v1:172.21.15.6:6789/0 2024-03-25T19:23:08.595 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.594+0000 7febacb04640 10 monclient(hunting): picked mon.noname-b con 0x7feba80b4ff0 addr v1:172.21.15.192:6789/0 2024-03-25T19:23:08.596 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.594+0000 7febacb04640 10 monclient(hunting): picked mon.noname-c con 0x7feba813c480 addr v1:172.21.15.6:6790/0 2024-03-25T19:23:08.596 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.594+0000 7febacb04640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba8143c80 con 0x7feba80b4670 2024-03-25T19:23:08.596 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.594+0000 7febacb04640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba8146570 con 0x7feba80b4ff0 2024-03-25T19:23:08.596 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.594+0000 7febacb04640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba8130cc0 con 0x7feba813c480 2024-03-25T19:23:08.596 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.594+0000 7febacb04640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:08.596 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.594+0000 7febacb04640 10 monclient(hunting): authenticate will time out at 6492.833998s 2024-03-25T19:23:08.599 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.599+0000 7feba659c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7feba80b4670 0x7feba8145750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46974/0 (socket says 172.21.15.6:46974) 2024-03-25T19:23:08.599 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.599+0000 7feba659c640 1 -- 172.21.15.6:0/3526246033 learned_addr learned my addr 172.21.15.6:0/3526246033 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:23:08.602 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.602+0000 7feb9effd640 10 client.?.objecter ms_handle_connect 0x7feba80b4670 2024-03-25T19:23:08.602 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.602+0000 7feb9effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:08.603 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.603+0000 7feb9effd640 10 client.?.objecter ms_handle_connect 0x7feba80b4ff0 2024-03-25T19:23:08.603 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.603+0000 7feb9effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:08.608 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.608+0000 7feb9effd640 1 -- 172.21.15.6:0/3526246033 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 62458601 0 0) 0x7feba8146570 con 0x7feba80b4ff0 2024-03-25T19:23:08.608 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.608+0000 7feb9effd640 10 monclient(hunting): _negotiate 2024-03-25T19:23:08.608 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.608+0000 7feb9effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:08.608 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.608+0000 7feb9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:08.608 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.608+0000 7feb9effd640 10 monclient(hunting): my global_id is 4639 2024-03-25T19:23:08.608 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.608+0000 7feb9effd640 1 -- 172.21.15.6:0/3526246033 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb88004ee0 con 0x7feba80b4ff0 2024-03-25T19:23:08.609 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.609+0000 7feb9effd640 10 client.?.objecter ms_handle_connect 0x7feba813c480 2024-03-25T19:23:08.609 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.609+0000 7feb9effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:08.612 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.612+0000 7feb9effd640 1 -- 172.21.15.6:0/3526246033 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1944852779 0 0) 0x7feba8143c80 con 0x7feba80b4670 2024-03-25T19:23:08.612 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.612+0000 7feb9effd640 10 monclient(hunting): _negotiate 2024-03-25T19:23:08.612 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.612+0000 7feb9effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:08.612 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.612+0000 7feb9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:08.612 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.612+0000 7feb9effd640 10 monclient(hunting): my global_id is 4629 2024-03-25T19:23:08.612 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.612+0000 7feb9effd640 1 -- 172.21.15.6:0/3526246033 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feba8146570 con 0x7feba80b4670 2024-03-25T19:23:08.614 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.614+0000 7feb9effd640 1 -- 172.21.15.6:0/3526246033 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3703378000 0 0) 0x7feb88004ee0 con 0x7feba80b4ff0 2024-03-25T19:23:08.614 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.614+0000 7feb9effd640 1 -- 172.21.15.6:0/3526246033 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feba8143c80 con 0x7feba80b4ff0 2024-03-25T19:23:08.614 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.614+0000 7feb9effd640 1 -- 172.21.15.6:0/3526246033 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7feb90001930 con 0x7feba80b4ff0 2024-03-25T19:23:08.614 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.614+0000 7feb9effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:08.614 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.614+0000 7feb9effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:23:08.614 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.614+0000 7feb9effd640 10 monclient(hunting): dump: 2024-03-25T19:23:08.614 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:23:08.614 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:08.614 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:08.614 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:08.614 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:08.615 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:08.615 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:23:08.615 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:23:08.615 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:23:08.615 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:08.615 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.614+0000 7feb9effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:23:08.619 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.619+0000 7feb9effd640 1 -- 172.21.15.6:0/3526246033 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1847402984 0 0) 0x7feba8146570 con 0x7feba80b4670 2024-03-25T19:23:08.619 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.619+0000 7feb9effd640 1 -- 172.21.15.6:0/3526246033 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb88004ee0 con 0x7feba80b4670 2024-03-25T19:23:08.619 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.619+0000 7feb9effd640 1 -- 172.21.15.6:0/3526246033 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7feba0001ea0 con 0x7feba80b4670 2024-03-25T19:23:08.619 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.619+0000 7feb9effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:08.619 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.619+0000 7feb9effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:23:08.619 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.619+0000 7feb9effd640 10 monclient(hunting): dump: 2024-03-25T19:23:08.619 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:23:08.619 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:08.620 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:08.620 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:08.620 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:08.620 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:08.620 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:23:08.620 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:23:08.620 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:23:08.620 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:08.621 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.621+0000 7feb9effd640 1 -- 172.21.15.6:0/3526246033 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1045479560 0 0) 0x7feba8143c80 con 0x7feba80b4ff0 2024-03-25T19:23:08.621 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.621+0000 7feb9effd640 1 -- 172.21.15.6:0/3526246033 >> v1:172.21.15.6:6790/0 conn(0x7feba813c480 legacy=0x7feba81e2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:08.621 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.621+0000 7feb9effd640 1 -- 172.21.15.6:0/3526246033 >> v1:172.21.15.6:6789/0 conn(0x7feba80b4670 legacy=0x7feba8145750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:08.621 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.621+0000 7feb9effd640 10 monclient: _finish_hunting 0 2024-03-25T19:23:08.622 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.621+0000 7feb9effd640 1 monclient: found mon.b 2024-03-25T19:23:08.622 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.621+0000 7feb9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:08.622 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.621+0000 7feb9effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:23:08.622 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.621+0000 7feb9effd640 1 -- 172.21.15.6:0/3526246033 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba8144a00 con 0x7feba80b4ff0 2024-03-25T19:23:08.622 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.621+0000 7feb9effd640 10 monclient: _finish_auth 0 2024-03-25T19:23:08.622 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.621+0000 7feb9effd640 10 monclient: _check_auth_tickets 2024-03-25T19:23:08.622 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.621+0000 7feb9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:08.622 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.621+0000 7febacb04640 5 monclient: authenticate success, global_id 4639 2024-03-25T19:23:08.622 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.621+0000 7febacb04640 10 monclient: _renew_subs 2024-03-25T19:23:08.622 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.621+0000 7febacb04640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:23:08.622 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.621+0000 7febacb04640 1 -- 172.21.15.6:0/3526246033 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feba8144ce0 con 0x7feba80b4ff0 2024-03-25T19:23:08.622 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.621+0000 7febacb04640 10 client.4639.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:23:08.623 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.621+0000 7febacb04640 10 monclient: _renew_subs 2024-03-25T19:23:08.623 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.621+0000 7febacb04640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:23:08.623 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.621+0000 7febacb04640 1 -- 172.21.15.6:0/3526246033 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feba813eec0 con 0x7feba80b4ff0 2024-03-25T19:23:08.623 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.621+0000 7febacb04640 1 librados: init done 2024-03-25T19:23:08.623 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.622+0000 7febacb04640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:08.623 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.622+0000 7febacb04640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:08.624 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.622+0000 7febacb04640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:23:08.624 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.622+0000 7febacb04640 1 -- 172.21.15.6:0/3526246033 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feba806dbd0 con 0x7feba80b4ff0 2024-03-25T19:23:08.909 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.908+0000 7feb9effd640 1 -- 172.21.15.6:0/3526246033 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb900035f0 con 0x7feba80b4ff0 2024-03-25T19:23:08.909 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.908+0000 7feb9effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:08.909 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.908+0000 7feb9effd640 1 -- 172.21.15.6:0/3526246033 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7feb90003900 con 0x7feba80b4ff0 2024-03-25T19:23:08.909 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.908+0000 7feb9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:08.909 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.908+0000 7feb9effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:23:08.909 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.908+0000 7feb9effd640 10 monclient: dump: 2024-03-25T19:23:08.909 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:23:08.909 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:08.909 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:08.909 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:08.910 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:08.910 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:08.910 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:23:08.910 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:23:08.910 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:23:08.910 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:08.910 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.908+0000 7feba4d99640 4 set_mon_vals no callback set 2024-03-25T19:23:08.910 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.909+0000 7feb9effd640 1 -- 172.21.15.6:0/3526246033 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7feb900198d0 con 0x7feba80b4ff0 2024-03-25T19:23:08.911 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.909+0000 7feba659c640 10 monclient: get_auth_request con 0x7feb8806de70 auth_method 0 2024-03-25T19:23:08.911 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.910+0000 7feb9effd640 10 client.4639.objecter ms_handle_connect 0x7feb8806de70 2024-03-25T19:23:08.917 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.917+0000 7feb9effd640 1 -- 172.21.15.6:0/3526246033 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7feb90053650 con 0x7feba80b4ff0 2024-03-25T19:23:08.917 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.917+0000 7feb9effd640 10 client.4639.objecter ms_dispatch 0x7feba8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:23:08.917 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.917+0000 7feb9effd640 3 client.4639.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:23:08.917 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.917+0000 7feb9effd640 3 client.4639.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:23:08.917 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.917+0000 7feb9effd640 20 client.4639.objecter dump_active .. 0 homeless 2024-03-25T19:23:08.918 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.917+0000 7feb9effd640 1 -- 172.21.15.6:0/3526246033 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feb9004ef90 con 0x7feba80b4ff0 2024-03-25T19:23:08.918 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.917+0000 7feb9effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:08.918 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:08.917+0000 7feb9effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:23:09.195 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:09.195+0000 7febacb04640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:09.195 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:09.195+0000 7febacb04640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:09.195 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:09.195+0000 7febacb04640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:23:09.195 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:09.195+0000 7febacb04640 1 -- 172.21.15.6:0/3526246033 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feba813d900 con 0x7feba80b4ff0 2024-03-25T19:23:09.209 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:09:26.556 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:09.256 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:09:26.602 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:09.280 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:09:26.626 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:09.318 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:09:26.636 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:09.324 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:09:26.671 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:09.324 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:09.324+0000 7feb9effd640 1 -- 172.21.15.6:0/3526246033 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7feb90051090 con 0x7feba80b4ff0 2024-03-25T19:23:09.325 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:09.324+0000 7feb9effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:09.325 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:09.324+0000 7feb9effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:23:09.325 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:23:09.325 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:23:09.325 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:09:26.643 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:09.327 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:09.327+0000 7febacb04640 10 librados: watch_flush enter 2024-03-25T19:23:09.327 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:09.327+0000 7febacb04640 10 librados: watch_flush exit 2024-03-25T19:23:09.327 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:09.327+0000 7febacb04640 20 client.4639.objecter shutdown clearing up homeless session... 2024-03-25T19:23:09.327 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:09.327+0000 7febacb04640 10 client.4639.objecter successfully canceled tick 2024-03-25T19:23:09.327 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:09.327+0000 7febacb04640 1 -- 172.21.15.6:0/3526246033 >> v1:172.21.15.6:6800/2936652650 conn(0x7feb8806de70 legacy=0x7feb88070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:09.327 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:09.327+0000 7febacb04640 10 monclient: shutdown 2024-03-25T19:23:09.327 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:09.327+0000 7febacb04640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:09.327 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:09.327+0000 7febacb04640 1 -- 172.21.15.6:0/3526246033 >> v1:172.21.15.192:6789/0 conn(0x7feba80b4ff0 legacy=0x7feba8145e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:09.327 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:09.327+0000 7febacb04640 1 -- 172.21.15.6:0/3526246033 shutdown_connections 2024-03-25T19:23:09.328 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:09.327+0000 7febacb04640 1 -- 172.21.15.6:0/3526246033 >> 172.21.15.6:0/3526246033 conn(0x7feba80617a0 msgr2=0x7feba80b2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:09.328 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:09.327+0000 7febacb04640 1 -- 172.21.15.6:0/3526246033 shutdown_connections 2024-03-25T19:23:09.328 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:09.327+0000 7febacb04640 1 -- 172.21.15.6:0/3526246033 wait complete. 2024-03-25T19:23:09.328 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:09.327+0000 7febacb04640 1 librados: shutdown 2024-03-25T19:23:09.329 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:09:26.676 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:09.338 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:23:09.353 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:09:26.700 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:09.357 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:09:26.675 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:16.144 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:23:16.255 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.255+0000 7fdcde014640 10 monclient: get_monmap_and_config 2024-03-25T19:23:16.255 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.255+0000 7fdcde014640 10 monclient: build_initial_monmap 2024-03-25T19:23:16.255 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.255+0000 7fdcde014640 10 monclient: monmap: 2024-03-25T19:23:16.256 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:23:16.256 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:16.256 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:23:16.256705+0000 2024-03-25T19:23:16.256 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:23:16.256705+0000 2024-03-25T19:23:16.256 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:16.256 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:16.256 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:23:16.256 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:23:16.256 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:23:16.256 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:16.258 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.256+0000 7fdcde014640 1 Processor -- start 2024-03-25T19:23:16.258 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.256+0000 7fdcde014640 1 -- start start 2024-03-25T19:23:16.258 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.256+0000 7fdcde014640 10 monclient: init 2024-03-25T19:23:16.258 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.257+0000 7fdcde014640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:16.258 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.257+0000 7fdcde014640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:23:16.258 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.257+0000 7fdcde014640 10 monclient(hunting): picked mon.noname-a con 0x7fdcd80b4ff0 addr v1:172.21.15.6:6789/0 2024-03-25T19:23:16.258 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.257+0000 7fdcde014640 10 monclient(hunting): picked mon.noname-c con 0x7fdcd80b4670 addr v1:172.21.15.192:6789/0 2024-03-25T19:23:16.258 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.257+0000 7fdcde014640 10 monclient(hunting): picked mon.noname-b con 0x7fdcd813c480 addr v1:172.21.15.6:6790/0 2024-03-25T19:23:16.258 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.257+0000 7fdcde014640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcd80ab0e0 con 0x7fdcd80b4ff0 2024-03-25T19:23:16.258 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.257+0000 7fdcde014640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcd80ab3c0 con 0x7fdcd80b4670 2024-03-25T19:23:16.259 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.257+0000 7fdcde014640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcd80a9f30 con 0x7fdcd813c480 2024-03-25T19:23:16.259 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.257+0000 7fdcde014640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:16.259 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.257+0000 7fdcde014640 10 monclient(hunting): authenticate will time out at 6500.497627s 2024-03-25T19:23:16.265 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.265+0000 7fdcd6ffd640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fdcd80b4670 0x7fdcd81425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:52324/0 (socket says 172.21.15.6:52324) 2024-03-25T19:23:16.265 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.265+0000 7fdcd6ffd640 1 -- 172.21.15.6:0/1931136081 learned_addr learned my addr 172.21.15.6:0/1931136081 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:23:16.272 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.272+0000 7fdcd67fc640 1 -- 172.21.15.6:0/1931136081 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2694574417 0 0) 0x7fdcd80ab3c0 con 0x7fdcd80b4670 2024-03-25T19:23:16.273 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.272+0000 7fdcd67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:23:16.273 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.272+0000 7fdcd67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:16.273 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.272+0000 7fdcd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:16.273 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.272+0000 7fdcd67fc640 10 monclient(hunting): my global_id is 4642 2024-03-25T19:23:16.273 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.272+0000 7fdcd67fc640 1 -- 172.21.15.6:0/1931136081 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcc0003520 con 0x7fdcd80b4670 2024-03-25T19:23:16.281 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.281+0000 7fdcd67fc640 1 -- 172.21.15.6:0/1931136081 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 141164718 0 0) 0x7fdcd80a9f30 con 0x7fdcd813c480 2024-03-25T19:23:16.281 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.281+0000 7fdcd67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:23:16.281 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.281+0000 7fdcd67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:16.281 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.281+0000 7fdcd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:16.281 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.281+0000 7fdcd67fc640 10 monclient(hunting): my global_id is 4634 2024-03-25T19:23:16.281 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.281+0000 7fdcd67fc640 1 -- 172.21.15.6:0/1931136081 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcd80ab3c0 con 0x7fdcd813c480 2024-03-25T19:23:16.283 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.283+0000 7fdcd67fc640 1 -- 172.21.15.6:0/1931136081 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 955912140 0 0) 0x7fdcc0003520 con 0x7fdcd80b4670 2024-03-25T19:23:16.283 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.283+0000 7fdcd67fc640 1 -- 172.21.15.6:0/1931136081 >> v1:172.21.15.6:6790/0 conn(0x7fdcd813c480 legacy=0x7fdcd814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:16.283 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.283+0000 7fdcd67fc640 1 -- 172.21.15.6:0/1931136081 >> v1:172.21.15.6:6789/0 conn(0x7fdcd80b4ff0 legacy=0x7fdcd813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:16.283 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.283+0000 7fdcd67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:23:16.283 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.283+0000 7fdcd67fc640 1 monclient: found mon.noname-c 2024-03-25T19:23:16.284 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.283+0000 7fdcd67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:16.284 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.283+0000 7fdcd67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:23:16.284 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.283+0000 7fdcd67fc640 1 -- 172.21.15.6:0/1931136081 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd80a9b70 con 0x7fdcd80b4670 2024-03-25T19:23:16.284 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.283+0000 7fdcd67fc640 10 monclient: _finish_auth 0 2024-03-25T19:23:16.284 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.283+0000 7fdcd67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:23:16.284 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.283+0000 7fdcd67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:16.284 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.283+0000 7fdcd67fc640 1 -- 172.21.15.6:0/1931136081 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fdccc001690 con 0x7fdcd80b4670 2024-03-25T19:23:16.284 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.283+0000 7fdcd67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:16.284 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.283+0000 7fdcd67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:23:16.284 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.283+0000 7fdcd67fc640 10 monclient: dump: 2024-03-25T19:23:16.284 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:23:16.284 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:16.284 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:16.284 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:16.284 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:16.285 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:16.285 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:23:16.285 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:23:16.285 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:23:16.285 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:16.285 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.283+0000 7fdcde014640 5 monclient: authenticate success, global_id 4642 2024-03-25T19:23:16.285 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.283+0000 7fdcde014640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:23:16.292 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.292+0000 7fdcd67fc640 1 -- 172.21.15.6:0/1931136081 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdccc0017f0 con 0x7fdcd80b4670 2024-03-25T19:23:16.292 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.292+0000 7fdcd67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:16.292 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.292+0000 7fdcd67fc640 1 -- 172.21.15.6:0/1931136081 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fdccc001c00 con 0x7fdcd80b4670 2024-03-25T19:23:16.292 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.292+0000 7fdcd67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:16.292 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.292+0000 7fdcd67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:23:16.292 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.292+0000 7fdcd67fc640 10 monclient: dump: 2024-03-25T19:23:16.292 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:23:16.293 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:16.293 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:16.293 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:16.293 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:16.293 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:16.293 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:23:16.293 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:23:16.293 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:23:16.293 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:16.293 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.292+0000 7fdcde014640 10 monclient: get_monmap_and_config success 2024-03-25T19:23:16.293 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.292+0000 7fdcde014640 4 set_mon_vals no callback set 2024-03-25T19:23:16.295 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.293+0000 7fdcde014640 10 monclient: shutdown 2024-03-25T19:23:16.296 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.293+0000 7fdcde014640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:16.296 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.293+0000 7fdcde014640 1 -- 172.21.15.6:0/1931136081 >> v1:172.21.15.192:6789/0 conn(0x7fdcd80b4670 legacy=0x7fdcd81425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:16.296 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.293+0000 7fdcde014640 1 -- 172.21.15.6:0/1931136081 shutdown_connections 2024-03-25T19:23:16.296 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.293+0000 7fdcde014640 1 -- 172.21.15.6:0/1931136081 >> 172.21.15.6:0/1931136081 conn(0x7fdcd80617a0 msgr2=0x7fdcd8061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:16.296 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.293+0000 7fdcde014640 1 -- 172.21.15.6:0/1931136081 shutdown_connections 2024-03-25T19:23:16.296 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.293+0000 7fdcde014640 1 -- 172.21.15.6:0/1931136081 wait complete. 2024-03-25T19:23:16.296 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.294+0000 7fdcde014640 10 monclient: build_initial_monmap 2024-03-25T19:23:16.296 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.294+0000 7fdcde014640 10 monclient: monmap: 2024-03-25T19:23:16.296 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:23:16.296 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:23:16.296 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:23:16.296 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:23:16.296 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:16.297 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:23:16.297 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:23:16.297 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:23:16.297 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:23:16.297 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:16.297 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.294+0000 7fdcde014640 1 librados: starting msgr at 2024-03-25T19:23:16.297 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.294+0000 7fdcde014640 1 librados: starting objecter 2024-03-25T19:23:16.297 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.294+0000 7fdcde014640 1 Processor -- start 2024-03-25T19:23:16.297 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.295+0000 7fdcde014640 1 -- start start 2024-03-25T19:23:16.297 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.295+0000 7fdcde014640 1 librados: setting wanted keys 2024-03-25T19:23:16.297 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.295+0000 7fdcde014640 1 librados: calling monclient init 2024-03-25T19:23:16.297 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.295+0000 7fdcde014640 10 monclient: init 2024-03-25T19:23:16.297 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.295+0000 7fdcde014640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:16.297 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.295+0000 7fdcde014640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:23:16.298 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.295+0000 7fdcde014640 10 monclient(hunting): picked mon.noname-c con 0x7fdcd80b4670 addr v1:172.21.15.6:6790/0 2024-03-25T19:23:16.298 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.295+0000 7fdcde014640 10 monclient(hunting): picked mon.noname-a con 0x7fdcd80b4ff0 addr v1:172.21.15.6:6789/0 2024-03-25T19:23:16.298 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.295+0000 7fdcde014640 10 monclient(hunting): picked mon.noname-b con 0x7fdcd813c480 addr v1:172.21.15.192:6789/0 2024-03-25T19:23:16.298 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.295+0000 7fdcde014640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcd80b05a0 con 0x7fdcd80b4ff0 2024-03-25T19:23:16.298 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.295+0000 7fdcde014640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcd80b0880 con 0x7fdcd813c480 2024-03-25T19:23:16.298 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.295+0000 7fdcde014640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcd80b02c0 con 0x7fdcd80b4670 2024-03-25T19:23:16.298 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.295+0000 7fdcde014640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:16.298 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.295+0000 7fdcde014640 10 monclient(hunting): authenticate will time out at 6500.535585s 2024-03-25T19:23:16.302 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.302+0000 7fdcd6ffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fdcd80b4ff0 0x7fdcd80ae600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51526/0 (socket says 172.21.15.6:51526) 2024-03-25T19:23:16.302 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.302+0000 7fdcd6ffd640 1 -- 172.21.15.6:0/2039732568 learned_addr learned my addr 172.21.15.6:0/2039732568 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:23:16.303 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.303+0000 7fdcbffff640 10 client.?.objecter ms_handle_connect 0x7fdcd80b4ff0 2024-03-25T19:23:16.303 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.303+0000 7fdcbffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:16.312 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.312+0000 7fdcbffff640 10 client.?.objecter ms_handle_connect 0x7fdcd80b4670 2024-03-25T19:23:16.312 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.312+0000 7fdcbffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:16.316 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.316+0000 7fdcbffff640 10 client.?.objecter ms_handle_connect 0x7fdcd813c480 2024-03-25T19:23:16.316 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.316+0000 7fdcbffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:16.318 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.319+0000 7fdcbffff640 1 -- 172.21.15.6:0/2039732568 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2486291711 0 0) 0x7fdcd80b02c0 con 0x7fdcd80b4670 2024-03-25T19:23:16.319 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.319+0000 7fdcbffff640 10 monclient(hunting): _negotiate 2024-03-25T19:23:16.319 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.319+0000 7fdcbffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:16.319 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.319+0000 7fdcbffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:16.319 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.319+0000 7fdcbffff640 10 monclient(hunting): my global_id is 4637 2024-03-25T19:23:16.319 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.319+0000 7fdcbffff640 1 -- 172.21.15.6:0/2039732568 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcc4004eb0 con 0x7fdcd80b4670 2024-03-25T19:23:16.320 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.319+0000 7fdcbffff640 1 -- 172.21.15.6:0/2039732568 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3722918433 0 0) 0x7fdcd80b05a0 con 0x7fdcd80b4ff0 2024-03-25T19:23:16.320 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.319+0000 7fdcbffff640 10 monclient(hunting): _negotiate 2024-03-25T19:23:16.320 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.319+0000 7fdcbffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:16.320 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.319+0000 7fdcbffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:16.320 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.319+0000 7fdcbffff640 10 monclient(hunting): my global_id is 4635 2024-03-25T19:23:16.320 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.319+0000 7fdcbffff640 1 -- 172.21.15.6:0/2039732568 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcd80b02c0 con 0x7fdcd80b4ff0 2024-03-25T19:23:16.327 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.327+0000 7fdcbffff640 1 -- 172.21.15.6:0/2039732568 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3659136473 0 0) 0x7fdcd80b0880 con 0x7fdcd813c480 2024-03-25T19:23:16.327 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.327+0000 7fdcbffff640 10 monclient(hunting): _negotiate 2024-03-25T19:23:16.327 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.327+0000 7fdcbffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:16.327 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.327+0000 7fdcbffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:16.327 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.327+0000 7fdcbffff640 10 monclient(hunting): my global_id is 4645 2024-03-25T19:23:16.327 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.327+0000 7fdcbffff640 1 -- 172.21.15.6:0/2039732568 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcd80b05a0 con 0x7fdcd813c480 2024-03-25T19:23:16.336 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.336+0000 7fdcbffff640 1 -- 172.21.15.6:0/2039732568 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3276951322 0 0) 0x7fdcc4004eb0 con 0x7fdcd80b4670 2024-03-25T19:23:16.336 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.336+0000 7fdcbffff640 1 -- 172.21.15.6:0/2039732568 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdcd80b0880 con 0x7fdcd80b4670 2024-03-25T19:23:16.336 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.336+0000 7fdcbffff640 1 -- 172.21.15.6:0/2039732568 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fdcc8001930 con 0x7fdcd80b4670 2024-03-25T19:23:16.336 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.336+0000 7fdcbffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:16.337 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.336+0000 7fdcbffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:23:16.337 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.336+0000 7fdcbffff640 10 monclient(hunting): dump: 2024-03-25T19:23:16.337 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:23:16.337 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:16.337 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:16.337 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:16.337 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:16.337 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:16.337 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:23:16.337 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:23:16.337 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:23:16.337 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:16.337 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.336+0000 7fdcbffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:23:16.337 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.336+0000 7fdcbffff640 1 -- 172.21.15.6:0/2039732568 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 715885213 0 0) 0x7fdcd80b05a0 con 0x7fdcd813c480 2024-03-25T19:23:16.337 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.336+0000 7fdcbffff640 1 -- 172.21.15.6:0/2039732568 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdcc4004eb0 con 0x7fdcd813c480 2024-03-25T19:23:16.338 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.336+0000 7fdcbffff640 1 -- 172.21.15.6:0/2039732568 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fdcd0001bd0 con 0x7fdcd813c480 2024-03-25T19:23:16.338 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.336+0000 7fdcbffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:16.338 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.336+0000 7fdcbffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:23:16.338 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.336+0000 7fdcbffff640 10 monclient(hunting): dump: 2024-03-25T19:23:16.338 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:23:16.338 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:16.338 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:16.338 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:16.338 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:16.338 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:16.338 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:23:16.338 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:23:16.338 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:23:16.338 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:16.340 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.339+0000 7fdcbffff640 1 -- 172.21.15.6:0/2039732568 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3376972635 0 0) 0x7fdcc4004eb0 con 0x7fdcd813c480 2024-03-25T19:23:16.340 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.339+0000 7fdcbffff640 1 -- 172.21.15.6:0/2039732568 >> v1:172.21.15.6:6790/0 conn(0x7fdcd80b4670 legacy=0x7fdcd80b0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:16.341 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.339+0000 7fdcbffff640 1 -- 172.21.15.6:0/2039732568 >> v1:172.21.15.6:6789/0 conn(0x7fdcd80b4ff0 legacy=0x7fdcd80ae600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:16.341 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.339+0000 7fdcbffff640 10 monclient: _finish_hunting 0 2024-03-25T19:23:16.341 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.339+0000 7fdcbffff640 1 monclient: found mon.b 2024-03-25T19:23:16.341 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.339+0000 7fdcbffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:16.341 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.339+0000 7fdcbffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:23:16.341 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.339+0000 7fdcbffff640 1 -- 172.21.15.6:0/2039732568 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd80b13b0 con 0x7fdcd813c480 2024-03-25T19:23:16.341 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.339+0000 7fdcbffff640 10 monclient: _finish_auth 0 2024-03-25T19:23:16.341 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.339+0000 7fdcbffff640 10 monclient: _check_auth_tickets 2024-03-25T19:23:16.341 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.339+0000 7fdcbffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:16.341 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.339+0000 7fdcde014640 5 monclient: authenticate success, global_id 4645 2024-03-25T19:23:16.341 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.339+0000 7fdcde014640 10 monclient: _renew_subs 2024-03-25T19:23:16.341 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.339+0000 7fdcde014640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:23:16.341 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.339+0000 7fdcde014640 1 -- 172.21.15.6:0/2039732568 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcd80b1660 con 0x7fdcd813c480 2024-03-25T19:23:16.341 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.339+0000 7fdcde014640 10 client.4645.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:23:16.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.339+0000 7fdcde014640 10 monclient: _renew_subs 2024-03-25T19:23:16.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.339+0000 7fdcde014640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:23:16.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.339+0000 7fdcde014640 1 -- 172.21.15.6:0/2039732568 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcd81e3780 con 0x7fdcd813c480 2024-03-25T19:23:16.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.339+0000 7fdcde014640 1 librados: init done 2024-03-25T19:23:16.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.340+0000 7fdcde014640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:16.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.340+0000 7fdcde014640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:16.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.340+0000 7fdcde014640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:23:16.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.340+0000 7fdcde014640 1 -- 172.21.15.6:0/2039732568 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcd8076320 con 0x7fdcd813c480 2024-03-25T19:23:16.610 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.609+0000 7fdcbffff640 1 -- 172.21.15.6:0/2039732568 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdcd0001fe0 con 0x7fdcd813c480 2024-03-25T19:23:16.610 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.609+0000 7fdcbffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:16.610 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.609+0000 7fdcbffff640 1 -- 172.21.15.6:0/2039732568 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fdcd0003b10 con 0x7fdcd813c480 2024-03-25T19:23:16.610 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.609+0000 7fdcbffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:16.610 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.609+0000 7fdcbffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:23:16.610 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.609+0000 7fdcbffff640 10 monclient: dump: 2024-03-25T19:23:16.610 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:23:16.610 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:16.610 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:16.611 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:16.611 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:16.611 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:16.611 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:23:16.611 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:23:16.611 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:23:16.611 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:16.611 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.609+0000 7fdcd5ffb640 4 set_mon_vals no callback set 2024-03-25T19:23:16.611 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.610+0000 7fdcbffff640 1 -- 172.21.15.6:0/2039732568 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7fdcd0019ae0 con 0x7fdcd813c480 2024-03-25T19:23:16.611 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.611+0000 7fdcd77fe640 10 monclient: get_auth_request con 0x7fdcc406df70 auth_method 0 2024-03-25T19:23:16.612 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.611+0000 7fdcbffff640 10 client.4645.objecter ms_handle_connect 0x7fdcc406df70 2024-03-25T19:23:16.615 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.615+0000 7fdcbffff640 1 -- 172.21.15.6:0/2039732568 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7fdcd0053880 con 0x7fdcd813c480 2024-03-25T19:23:16.615 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.615+0000 7fdcbffff640 10 client.4645.objecter ms_dispatch 0x7fdcd8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:23:16.616 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.615+0000 7fdcbffff640 3 client.4645.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:23:16.616 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.615+0000 7fdcbffff640 3 client.4645.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:23:16.616 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.615+0000 7fdcbffff640 20 client.4645.objecter dump_active .. 0 homeless 2024-03-25T19:23:16.616 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.615+0000 7fdcbffff640 1 -- 172.21.15.6:0/2039732568 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdcd004f1c0 con 0x7fdcd813c480 2024-03-25T19:23:16.616 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.616+0000 7fdcbffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:16.616 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.616+0000 7fdcbffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:23:16.850 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.850+0000 7fdcde014640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:16.850 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.850+0000 7fdcde014640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:16.850 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.850+0000 7fdcde014640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:23:16.850 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.850+0000 7fdcde014640 1 -- 172.21.15.6:0/2039732568 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdcd806dbd0 con 0x7fdcd813c480 2024-03-25T19:23:16.876 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.876+0000 7fdcbffff640 1 -- 172.21.15.6:0/2039732568 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7fdcd00512c0 con 0x7fdcd813c480 2024-03-25T19:23:16.876 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.876+0000 7fdcbffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:16.876 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.876+0000 7fdcbffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:23:16.876 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:23:16.877 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:23:16.880 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.879+0000 7fdcde014640 10 librados: watch_flush enter 2024-03-25T19:23:16.880 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.879+0000 7fdcde014640 10 librados: watch_flush exit 2024-03-25T19:23:16.880 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.879+0000 7fdcde014640 20 client.4645.objecter shutdown clearing up homeless session... 2024-03-25T19:23:16.880 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.879+0000 7fdcde014640 10 client.4645.objecter successfully canceled tick 2024-03-25T19:23:16.880 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.879+0000 7fdcde014640 1 -- 172.21.15.6:0/2039732568 >> v1:172.21.15.6:6800/2936652650 conn(0x7fdcc406df70 legacy=0x7fdcc4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:16.880 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.879+0000 7fdcde014640 10 monclient: shutdown 2024-03-25T19:23:16.880 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.879+0000 7fdcde014640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:16.880 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.879+0000 7fdcde014640 1 -- 172.21.15.6:0/2039732568 >> v1:172.21.15.192:6789/0 conn(0x7fdcd813c480 legacy=0x7fdcd80aed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:16.880 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.879+0000 7fdcde014640 1 -- 172.21.15.6:0/2039732568 shutdown_connections 2024-03-25T19:23:16.880 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.879+0000 7fdcde014640 1 -- 172.21.15.6:0/2039732568 >> 172.21.15.6:0/2039732568 conn(0x7fdcd80617a0 msgr2=0x7fdcd80b6e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:16.880 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.879+0000 7fdcde014640 1 -- 172.21.15.6:0/2039732568 shutdown_connections 2024-03-25T19:23:16.880 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.879+0000 7fdcde014640 1 -- 172.21.15.6:0/2039732568 wait complete. 2024-03-25T19:23:16.880 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:16.879+0000 7fdcde014640 1 librados: shutdown 2024-03-25T19:23:16.891 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:23:23.697 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:23:23.815 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.815+0000 7f915ad13640 10 monclient: get_monmap_and_config 2024-03-25T19:23:23.815 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.815+0000 7f915ad13640 10 monclient: build_initial_monmap 2024-03-25T19:23:23.815 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.815+0000 7f915ad13640 10 monclient: monmap: 2024-03-25T19:23:23.815 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:23:23.815 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:23.815 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:23:23.816280+0000 2024-03-25T19:23:23.816 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:23:23.816280+0000 2024-03-25T19:23:23.816 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:23.816 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:23.816 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:23:23.816 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:23:23.816 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:23:23.816 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:23.817 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.816+0000 7f915ad13640 1 Processor -- start 2024-03-25T19:23:23.818 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.816+0000 7f915ad13640 1 -- start start 2024-03-25T19:23:23.818 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.816+0000 7f915ad13640 10 monclient: init 2024-03-25T19:23:23.818 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.816+0000 7f915ad13640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:23.818 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.816+0000 7f915ad13640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:23:23.818 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.817+0000 7f915ad13640 10 monclient(hunting): picked mon.noname-c con 0x7f91540b4ff0 addr v1:172.21.15.192:6789/0 2024-03-25T19:23:23.818 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.817+0000 7f915ad13640 10 monclient(hunting): picked mon.noname-b con 0x7f91540b4670 addr v1:172.21.15.6:6790/0 2024-03-25T19:23:23.818 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.817+0000 7f915ad13640 10 monclient(hunting): picked mon.noname-a con 0x7f915413c480 addr v1:172.21.15.6:6789/0 2024-03-25T19:23:23.818 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.817+0000 7f915ad13640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91540ab0e0 con 0x7f915413c480 2024-03-25T19:23:23.818 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.817+0000 7f915ad13640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91540ab3c0 con 0x7f91540b4ff0 2024-03-25T19:23:23.818 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.817+0000 7f915ad13640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91540a9f30 con 0x7f91540b4670 2024-03-25T19:23:23.818 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.817+0000 7f915ad13640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:23.818 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.817+0000 7f915ad13640 10 monclient(hunting): authenticate will time out at 6508.056873s 2024-03-25T19:23:23.830 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.830+0000 7f9153fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f91540b4670 0x7f91541425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:55108/0 (socket says 172.21.15.6:55108) 2024-03-25T19:23:23.830 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.830+0000 7f9153fff640 1 -- 172.21.15.6:0/2487862177 learned_addr learned my addr 172.21.15.6:0/2487862177 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:23:23.845 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.845+0000 7f91537fe640 1 -- 172.21.15.6:0/2487862177 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3306911473 0 0) 0x7f91540ab3c0 con 0x7f91540b4ff0 2024-03-25T19:23:23.845 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.845+0000 7f91537fe640 10 monclient(hunting): _negotiate 2024-03-25T19:23:23.845 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.845+0000 7f91537fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:23.845 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.845+0000 7f91537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:23.845 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.845+0000 7f91537fe640 10 monclient(hunting): my global_id is 4648 2024-03-25T19:23:23.846 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.845+0000 7f91537fe640 1 -- 172.21.15.6:0/2487862177 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9144003520 con 0x7f91540b4ff0 2024-03-25T19:23:23.849 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.849+0000 7f91537fe640 1 -- 172.21.15.6:0/2487862177 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3798197314 0 0) 0x7f91540a9f30 con 0x7f91540b4670 2024-03-25T19:23:23.849 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.849+0000 7f91537fe640 10 monclient(hunting): _negotiate 2024-03-25T19:23:23.849 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.849+0000 7f91537fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:23.849 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.849+0000 7f91537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:23.849 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.849+0000 7f91537fe640 10 monclient(hunting): my global_id is 4640 2024-03-25T19:23:23.849 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.849+0000 7f91537fe640 1 -- 172.21.15.6:0/2487862177 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91540ab3c0 con 0x7f91540b4670 2024-03-25T19:23:23.861 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.861+0000 7f91537fe640 1 -- 172.21.15.6:0/2487862177 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1955980598 0 0) 0x7f91540ab3c0 con 0x7f91540b4670 2024-03-25T19:23:23.861 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.861+0000 7f91537fe640 1 -- 172.21.15.6:0/2487862177 >> v1:172.21.15.192:6789/0 conn(0x7f91540b4ff0 legacy=0x7f915413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:23.861 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.861+0000 7f91537fe640 1 -- 172.21.15.6:0/2487862177 >> v1:172.21.15.6:6789/0 conn(0x7f915413c480 legacy=0x7f915414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:23.861 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.861+0000 7f91537fe640 10 monclient: _finish_hunting 0 2024-03-25T19:23:23.861 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.861+0000 7f91537fe640 1 monclient: found mon.noname-b 2024-03-25T19:23:23.862 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.861+0000 7f91537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:23.862 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.861+0000 7f91537fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-03-25T19:23:23.862 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.861+0000 7f91537fe640 1 -- 172.21.15.6:0/2487862177 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91540a9b70 con 0x7f91540b4670 2024-03-25T19:23:23.862 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.861+0000 7f91537fe640 10 monclient: _finish_auth 0 2024-03-25T19:23:23.862 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.861+0000 7f91537fe640 10 monclient: _check_auth_tickets 2024-03-25T19:23:23.862 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.861+0000 7f91537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:23.862 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.861+0000 7f91537fe640 1 -- 172.21.15.6:0/2487862177 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f9140001670 con 0x7f91540b4670 2024-03-25T19:23:23.862 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.861+0000 7f91537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:23.862 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.861+0000 7f91537fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:23:23.862 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.861+0000 7f91537fe640 10 monclient: dump: 2024-03-25T19:23:23.862 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:23:23.862 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:23.863 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:23.863 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:23.863 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:23.863 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:23.863 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:23:23.863 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:23:23.863 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:23:23.863 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:23.863 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.861+0000 7f915ad13640 5 monclient: authenticate success, global_id 4640 2024-03-25T19:23:23.863 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.861+0000 7f915ad13640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:23:23.880 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.880+0000 7f91537fe640 1 -- 172.21.15.6:0/2487862177 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91400017d0 con 0x7f91540b4670 2024-03-25T19:23:23.880 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.880+0000 7f91537fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:23.880 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.880+0000 7f91537fe640 1 -- 172.21.15.6:0/2487862177 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f9140001ba0 con 0x7f91540b4670 2024-03-25T19:23:23.880 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.880+0000 7f91537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:23.880 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.880+0000 7f91537fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:23:23.880 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.880+0000 7f91537fe640 10 monclient: dump: 2024-03-25T19:23:23.880 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:23:23.880 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:23.880 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:23.881 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:23.881 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:23.881 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:23.881 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:23:23.881 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:23:23.881 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:23:23.881 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:23.881 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.880+0000 7f915ad13640 10 monclient: get_monmap_and_config success 2024-03-25T19:23:23.881 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.880+0000 7f915ad13640 4 set_mon_vals no callback set 2024-03-25T19:23:23.883 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.881+0000 7f915ad13640 10 monclient: shutdown 2024-03-25T19:23:23.883 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.881+0000 7f915ad13640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:23.883 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.881+0000 7f915ad13640 1 -- 172.21.15.6:0/2487862177 >> v1:172.21.15.6:6790/0 conn(0x7f91540b4670 legacy=0x7f91541425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:23.883 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.881+0000 7f915ad13640 1 -- 172.21.15.6:0/2487862177 shutdown_connections 2024-03-25T19:23:23.884 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.881+0000 7f915ad13640 1 -- 172.21.15.6:0/2487862177 >> 172.21.15.6:0/2487862177 conn(0x7f91540617a0 msgr2=0x7f9154061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:23.884 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.881+0000 7f915ad13640 1 -- 172.21.15.6:0/2487862177 shutdown_connections 2024-03-25T19:23:23.884 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.881+0000 7f915ad13640 1 -- 172.21.15.6:0/2487862177 wait complete. 2024-03-25T19:23:23.884 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.882+0000 7f915ad13640 10 monclient: build_initial_monmap 2024-03-25T19:23:23.884 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.882+0000 7f915ad13640 10 monclient: monmap: 2024-03-25T19:23:23.884 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:23:23.884 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:23:23.884 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:23:23.884 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:23:23.884 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:23.884 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:23:23.884 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:23:23.884 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:23:23.884 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:23:23.885 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:23.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.882+0000 7f915ad13640 1 librados: starting msgr at 2024-03-25T19:23:23.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.882+0000 7f915ad13640 1 librados: starting objecter 2024-03-25T19:23:23.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.882+0000 7f915ad13640 1 Processor -- start 2024-03-25T19:23:23.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.882+0000 7f915ad13640 1 -- start start 2024-03-25T19:23:23.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.882+0000 7f915ad13640 1 librados: setting wanted keys 2024-03-25T19:23:23.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.882+0000 7f915ad13640 1 librados: calling monclient init 2024-03-25T19:23:23.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.882+0000 7f915ad13640 10 monclient: init 2024-03-25T19:23:23.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.883+0000 7f915ad13640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:23.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.883+0000 7f915ad13640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:23:23.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.883+0000 7f915ad13640 10 monclient(hunting): picked mon.noname-c con 0x7f91540b4670 addr v1:172.21.15.6:6790/0 2024-03-25T19:23:23.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.883+0000 7f915ad13640 10 monclient(hunting): picked mon.noname-b con 0x7f91540b4ff0 addr v1:172.21.15.192:6789/0 2024-03-25T19:23:23.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.883+0000 7f915ad13640 10 monclient(hunting): picked mon.noname-a con 0x7f915413c480 addr v1:172.21.15.6:6789/0 2024-03-25T19:23:23.886 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.883+0000 7f915ad13640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9154143810 con 0x7f915413c480 2024-03-25T19:23:23.886 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.883+0000 7f915ad13640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f915413dd50 con 0x7f91540b4ff0 2024-03-25T19:23:23.886 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.883+0000 7f915ad13640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9154146490 con 0x7f91540b4670 2024-03-25T19:23:23.886 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.883+0000 7f915ad13640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:23.886 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.883+0000 7f915ad13640 10 monclient(hunting): authenticate will time out at 6508.123720s 2024-03-25T19:23:23.892 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.892+0000 7f9153fff640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f91540b4ff0 0x7f91541443b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:44148/0 (socket says 172.21.15.6:44148) 2024-03-25T19:23:23.892 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.892+0000 7f9153fff640 1 -- 172.21.15.6:0/2226899082 learned_addr learned my addr 172.21.15.6:0/2226899082 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:23:23.893 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.894+0000 7f91517fa640 10 client.?.objecter ms_handle_connect 0x7f91540b4ff0 2024-03-25T19:23:23.894 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.894+0000 7f91517fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:23.901 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.901+0000 7f91517fa640 10 client.?.objecter ms_handle_connect 0x7f915413c480 2024-03-25T19:23:23.901 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.901+0000 7f91517fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:23.906 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.905+0000 7f91517fa640 1 -- 172.21.15.6:0/2226899082 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3746546180 0 0) 0x7f915413dd50 con 0x7f91540b4ff0 2024-03-25T19:23:23.906 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.905+0000 7f91517fa640 10 monclient(hunting): _negotiate 2024-03-25T19:23:23.906 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.905+0000 7f91517fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:23.906 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.905+0000 7f91517fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:23.906 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.905+0000 7f91517fa640 10 monclient(hunting): my global_id is 4651 2024-03-25T19:23:23.906 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.905+0000 7f91517fa640 1 -- 172.21.15.6:0/2226899082 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9138004eb0 con 0x7f91540b4ff0 2024-03-25T19:23:23.906 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.906+0000 7f91517fa640 10 client.?.objecter ms_handle_connect 0x7f91540b4670 2024-03-25T19:23:23.906 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.906+0000 7f91517fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:23.915 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.915+0000 7f91517fa640 1 -- 172.21.15.6:0/2226899082 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1630029871 0 0) 0x7f9154146490 con 0x7f91540b4670 2024-03-25T19:23:23.915 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.915+0000 7f91517fa640 10 monclient(hunting): _negotiate 2024-03-25T19:23:23.915 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.915+0000 7f91517fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:23.915 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.915+0000 7f91517fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:23.915 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.915+0000 7f91517fa640 10 monclient(hunting): my global_id is 4643 2024-03-25T19:23:23.915 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.915+0000 7f91517fa640 1 -- 172.21.15.6:0/2226899082 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f915413dd50 con 0x7f91540b4670 2024-03-25T19:23:23.921 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.921+0000 7f91517fa640 1 -- 172.21.15.6:0/2226899082 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3759158082 0 0) 0x7f9154143810 con 0x7f915413c480 2024-03-25T19:23:23.921 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.921+0000 7f91517fa640 10 monclient(hunting): _negotiate 2024-03-25T19:23:23.921 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.921+0000 7f91517fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:23.921 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.921+0000 7f91517fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:23.921 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.921+0000 7f91517fa640 10 monclient(hunting): my global_id is 4641 2024-03-25T19:23:23.921 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.921+0000 7f91517fa640 1 -- 172.21.15.6:0/2226899082 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9154146490 con 0x7f915413c480 2024-03-25T19:23:23.926 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.926+0000 7f91517fa640 1 -- 172.21.15.6:0/2226899082 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1324883569 0 0) 0x7f9138004eb0 con 0x7f91540b4ff0 2024-03-25T19:23:23.926 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.926+0000 7f91517fa640 1 -- 172.21.15.6:0/2226899082 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9154143810 con 0x7f91540b4ff0 2024-03-25T19:23:23.926 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.926+0000 7f91517fa640 1 -- 172.21.15.6:0/2226899082 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f9140001ba0 con 0x7f91540b4ff0 2024-03-25T19:23:23.927 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.926+0000 7f91517fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:23.927 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.926+0000 7f91517fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:23:23.927 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.926+0000 7f91517fa640 10 monclient(hunting): dump: 2024-03-25T19:23:23.927 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:23:23.927 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:23.927 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:23.927 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:23.927 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:23.927 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:23.927 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:23:23.927 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:23:23.927 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:23:23.927 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:23.927 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.926+0000 7f91517fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:23:23.936 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.936+0000 7f91517fa640 1 -- 172.21.15.6:0/2226899082 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1410261087 0 0) 0x7f9154143810 con 0x7f91540b4ff0 2024-03-25T19:23:23.936 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.936+0000 7f91517fa640 1 -- 172.21.15.6:0/2226899082 >> v1:172.21.15.6:6790/0 conn(0x7f91540b4670 legacy=0x7f9154143ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:23.936 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.936+0000 7f91517fa640 1 -- 172.21.15.6:0/2226899082 >> v1:172.21.15.6:6789/0 conn(0x7f915413c480 legacy=0x7f91541e2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:23.936 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.936+0000 7f91517fa640 10 monclient: _finish_hunting 0 2024-03-25T19:23:23.937 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.936+0000 7f91517fa640 1 monclient: found mon.b 2024-03-25T19:23:23.937 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.936+0000 7f91517fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:23.937 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.936+0000 7f91517fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:23:23.937 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.936+0000 7f91517fa640 1 -- 172.21.15.6:0/2226899082 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9154145830 con 0x7f91540b4ff0 2024-03-25T19:23:23.937 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.936+0000 7f91517fa640 10 monclient: _finish_auth 0 2024-03-25T19:23:23.937 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.936+0000 7f91517fa640 10 monclient: _check_auth_tickets 2024-03-25T19:23:23.937 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.936+0000 7f91517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:23.937 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.936+0000 7f915ad13640 5 monclient: authenticate success, global_id 4651 2024-03-25T19:23:23.937 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.936+0000 7f915ad13640 10 monclient: _renew_subs 2024-03-25T19:23:23.937 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.936+0000 7f915ad13640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:23:23.937 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.936+0000 7f915ad13640 1 -- 172.21.15.6:0/2226899082 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9154145a20 con 0x7f91540b4ff0 2024-03-25T19:23:23.937 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.936+0000 7f915ad13640 10 client.4651.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:23:23.937 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.936+0000 7f915ad13640 10 monclient: _renew_subs 2024-03-25T19:23:23.938 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.936+0000 7f915ad13640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:23:23.938 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.936+0000 7f915ad13640 1 -- 172.21.15.6:0/2226899082 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f915413cd80 con 0x7f91540b4ff0 2024-03-25T19:23:23.938 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.936+0000 7f915ad13640 1 librados: init done 2024-03-25T19:23:23.938 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.937+0000 7f915ad13640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:23.939 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.937+0000 7f915ad13640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:23.939 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.937+0000 7f915ad13640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:23:23.939 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.937+0000 7f915ad13640 1 -- 172.21.15.6:0/2226899082 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91541435e0 con 0x7f91540b4ff0 2024-03-25T19:23:23.960 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.960+0000 7f91517fa640 1 -- 172.21.15.6:0/2226899082 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9140004a90 con 0x7f91540b4ff0 2024-03-25T19:23:23.960 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.960+0000 7f91517fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:23.960 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.960+0000 7f91517fa640 1 -- 172.21.15.6:0/2226899082 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f9140004d00 con 0x7f91540b4ff0 2024-03-25T19:23:23.960 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.960+0000 7f91517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:23.960 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.960+0000 7f91517fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:23:23.960 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.960+0000 7f91517fa640 10 monclient: dump: 2024-03-25T19:23:23.960 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:23:23.960 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:23.960 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:23.960 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:23.960 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:23.960 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:23.961 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:23:23.961 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:23:23.961 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:23:23.961 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:23.961 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:23.960+0000 7f9152ffd640 4 set_mon_vals no callback set 2024-03-25T19:23:24.231 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:24.231+0000 7f91517fa640 1 -- 172.21.15.6:0/2226899082 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f9140004e60 con 0x7f91540b4ff0 2024-03-25T19:23:24.231 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:24.231+0000 7f9158aaf640 10 monclient: get_auth_request con 0x7f913806de20 auth_method 0 2024-03-25T19:23:24.232 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:24.232+0000 7f91517fa640 10 client.4651.objecter ms_handle_connect 0x7f913806de20 2024-03-25T19:23:24.234 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:24.234+0000 7f91517fa640 1 -- 172.21.15.6:0/2226899082 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f9140054810 con 0x7f91540b4ff0 2024-03-25T19:23:24.234 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:24.234+0000 7f91517fa640 10 client.4651.objecter ms_dispatch 0x7f9154000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:23:24.234 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:24.234+0000 7f91517fa640 3 client.4651.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:23:24.234 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:24.234+0000 7f91517fa640 3 client.4651.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:23:24.234 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:24.234+0000 7f91517fa640 20 client.4651.objecter dump_active .. 0 homeless 2024-03-25T19:23:24.236 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:24.237+0000 7f91517fa640 1 -- 172.21.15.6:0/2226899082 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9140050320 con 0x7f91540b4ff0 2024-03-25T19:23:24.237 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:24.237+0000 7f91517fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:24.237 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:24.237+0000 7f91517fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:23:24.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:24.467+0000 7f915ad13640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:24.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:24.467+0000 7f915ad13640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:24.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:24.467+0000 7f915ad13640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:23:24.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:24.467+0000 7f915ad13640 1 -- 172.21.15.6:0/2226899082 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f91541439c0 con 0x7f91540b4ff0 2024-03-25T19:23:24.499 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:24.499+0000 7f91517fa640 1 -- 172.21.15.6:0/2226899082 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f91541439c0 con 0x7f91540b4ff0 2024-03-25T19:23:24.499 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:24.499+0000 7f91517fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:24.499 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:24.499+0000 7f91517fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:23:24.499 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:23:24.499 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:23:24.502 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:24.501+0000 7f915ad13640 10 librados: watch_flush enter 2024-03-25T19:23:24.502 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:24.501+0000 7f915ad13640 10 librados: watch_flush exit 2024-03-25T19:23:24.502 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:24.501+0000 7f915ad13640 20 client.4651.objecter shutdown clearing up homeless session... 2024-03-25T19:23:24.502 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:24.501+0000 7f915ad13640 10 client.4651.objecter successfully canceled tick 2024-03-25T19:23:24.502 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:24.502+0000 7f915ad13640 1 -- 172.21.15.6:0/2226899082 >> v1:172.21.15.6:6800/2936652650 conn(0x7f913806de20 legacy=0x7f91380702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:24.502 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:24.502+0000 7f915ad13640 10 monclient: shutdown 2024-03-25T19:23:24.503 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:24.502+0000 7f915ad13640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:24.503 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:24.502+0000 7f915ad13640 1 -- 172.21.15.6:0/2226899082 >> v1:172.21.15.192:6789/0 conn(0x7f91540b4ff0 legacy=0x7f91541443b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:24.503 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:24.502+0000 7f915ad13640 1 -- 172.21.15.6:0/2226899082 shutdown_connections 2024-03-25T19:23:24.503 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:24.502+0000 7f915ad13640 1 -- 172.21.15.6:0/2226899082 >> 172.21.15.6:0/2226899082 conn(0x7f91540617a0 msgr2=0x7f91541405c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:24.503 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:24.502+0000 7f915ad13640 1 -- 172.21.15.6:0/2226899082 shutdown_connections 2024-03-25T19:23:24.503 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:24.502+0000 7f915ad13640 1 -- 172.21.15.6:0/2226899082 wait complete. 2024-03-25T19:23:24.503 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:24.502+0000 7f915ad13640 1 librados: shutdown 2024-03-25T19:23:24.513 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:23:29.248 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:09:46.595 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:29.251 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:09:46.598 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:29.325 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:09:46.672 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:29.367 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:09:46.685 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:29.377 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:09:46.695 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:29.387 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:09:46.734 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:29.398 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:09:46.745 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:29.404 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:09:46.722 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:29.429 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:09:46.776 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:31.320 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:23:31.437 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.437+0000 7fd167190640 10 monclient: get_monmap_and_config 2024-03-25T19:23:31.437 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.437+0000 7fd167190640 10 monclient: build_initial_monmap 2024-03-25T19:23:31.438 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.437+0000 7fd167190640 10 monclient: monmap: 2024-03-25T19:23:31.438 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:23:31.438 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:31.438 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:23:31.438772+0000 2024-03-25T19:23:31.438 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:23:31.438772+0000 2024-03-25T19:23:31.438 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:31.438 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:31.438 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:23:31.438 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:23:31.438 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:23:31.438 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:31.439 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.438+0000 7fd167190640 1 Processor -- start 2024-03-25T19:23:31.440 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.438+0000 7fd167190640 1 -- start start 2024-03-25T19:23:31.440 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.438+0000 7fd167190640 10 monclient: init 2024-03-25T19:23:31.440 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.439+0000 7fd167190640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:31.440 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.439+0000 7fd167190640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:23:31.440 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.439+0000 7fd167190640 10 monclient(hunting): picked mon.noname-a con 0x7fd1600b4e70 addr v1:172.21.15.6:6789/0 2024-03-25T19:23:31.440 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.439+0000 7fd167190640 10 monclient(hunting): picked mon.noname-c con 0x7fd1600b44b0 addr v1:172.21.15.192:6789/0 2024-03-25T19:23:31.440 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.439+0000 7fd167190640 10 monclient(hunting): picked mon.noname-b con 0x7fd1600b1b00 addr v1:172.21.15.6:6790/0 2024-03-25T19:23:31.440 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.439+0000 7fd167190640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1600a96f0 con 0x7fd1600b4e70 2024-03-25T19:23:31.440 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.439+0000 7fd167190640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1600ab0e0 con 0x7fd1600b44b0 2024-03-25T19:23:31.440 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.439+0000 7fd167190640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd160145f70 con 0x7fd1600b1b00 2024-03-25T19:23:31.440 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.439+0000 7fd167190640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:31.440 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.439+0000 7fd167190640 10 monclient(hunting): authenticate will time out at 6515.679188s 2024-03-25T19:23:31.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.445+0000 7fd16572d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fd1600b1b00 0x7fd16014dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:41404/0 (socket says 172.21.15.6:41404) 2024-03-25T19:23:31.445 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.445+0000 7fd16572d640 1 -- 172.21.15.6:0/2103154527 learned_addr learned my addr 172.21.15.6:0/2103154527 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:23:31.459 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.459+0000 7fd15f7fe640 1 -- 172.21.15.6:0/2103154527 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1849699441 0 0) 0x7fd1600a96f0 con 0x7fd1600b4e70 2024-03-25T19:23:31.459 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.459+0000 7fd15f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:23:31.459 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.459+0000 7fd15f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:31.459 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.459+0000 7fd15f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:31.459 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.459+0000 7fd15f7fe640 10 monclient(hunting): my global_id is 4644 2024-03-25T19:23:31.459 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.459+0000 7fd15f7fe640 1 -- 172.21.15.6:0/2103154527 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd150003520 con 0x7fd1600b4e70 2024-03-25T19:23:31.460 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.460+0000 7fd15f7fe640 1 -- 172.21.15.6:0/2103154527 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4219518871 0 0) 0x7fd160145f70 con 0x7fd1600b1b00 2024-03-25T19:23:31.460 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.460+0000 7fd15f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:23:31.460 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.460+0000 7fd15f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:31.460 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.460+0000 7fd15f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:31.460 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.460+0000 7fd15f7fe640 10 monclient(hunting): my global_id is 4646 2024-03-25T19:23:31.461 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.460+0000 7fd15f7fe640 1 -- 172.21.15.6:0/2103154527 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1600a96f0 con 0x7fd1600b1b00 2024-03-25T19:23:31.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.470+0000 7fd15f7fe640 1 -- 172.21.15.6:0/2103154527 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1524564644 0 0) 0x7fd1600a96f0 con 0x7fd1600b1b00 2024-03-25T19:23:31.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.470+0000 7fd15f7fe640 1 -- 172.21.15.6:0/2103154527 >> v1:172.21.15.192:6789/0 conn(0x7fd1600b44b0 legacy=0x7fd160145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:31.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.470+0000 7fd15f7fe640 1 -- 172.21.15.6:0/2103154527 >> v1:172.21.15.6:6789/0 conn(0x7fd1600b4e70 legacy=0x7fd16013f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:31.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.470+0000 7fd15f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:23:31.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.470+0000 7fd15f7fe640 1 monclient: found mon.noname-b 2024-03-25T19:23:31.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.470+0000 7fd15f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:31.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.470+0000 7fd15f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-03-25T19:23:31.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.470+0000 7fd15f7fe640 1 -- 172.21.15.6:0/2103154527 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1600a9b80 con 0x7fd1600b1b00 2024-03-25T19:23:31.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.470+0000 7fd15f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:23:31.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.470+0000 7fd15f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:23:31.471 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.470+0000 7fd15f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:31.472 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.470+0000 7fd15f7fe640 1 -- 172.21.15.6:0/2103154527 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fd1580019f0 con 0x7fd1600b1b00 2024-03-25T19:23:31.472 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.470+0000 7fd167190640 5 monclient: authenticate success, global_id 4646 2024-03-25T19:23:31.472 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.470+0000 7fd15f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:31.472 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.470+0000 7fd15f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:23:31.472 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.470+0000 7fd15f7fe640 10 monclient: dump: 2024-03-25T19:23:31.472 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:23:31.472 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:31.472 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:31.472 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:31.472 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:31.472 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:31.472 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:23:31.472 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:23:31.472 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:23:31.472 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:31.473 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.470+0000 7fd167190640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:23:31.503 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.503+0000 7fd15f7fe640 1 -- 172.21.15.6:0/2103154527 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd158001b50 con 0x7fd1600b1b00 2024-03-25T19:23:31.503 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.503+0000 7fd15f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:31.503 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.503+0000 7fd15f7fe640 1 -- 172.21.15.6:0/2103154527 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fd158001f20 con 0x7fd1600b1b00 2024-03-25T19:23:31.503 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.503+0000 7fd167190640 10 monclient: get_monmap_and_config success 2024-03-25T19:23:31.503 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.503+0000 7fd167190640 4 set_mon_vals no callback set 2024-03-25T19:23:31.503 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.503+0000 7fd15f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:31.503 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.503+0000 7fd15f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:23:31.503 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.503+0000 7fd15f7fe640 10 monclient: dump: 2024-03-25T19:23:31.503 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:23:31.503 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:31.503 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:31.503 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:31.503 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:31.504 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:31.504 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:23:31.504 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:23:31.504 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:23:31.504 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:31.505 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.504+0000 7fd167190640 10 monclient: shutdown 2024-03-25T19:23:31.505 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.504+0000 7fd167190640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:31.505 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.504+0000 7fd167190640 1 -- 172.21.15.6:0/2103154527 >> v1:172.21.15.6:6790/0 conn(0x7fd1600b1b00 legacy=0x7fd16014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:31.505 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.504+0000 7fd167190640 1 -- 172.21.15.6:0/2103154527 shutdown_connections 2024-03-25T19:23:31.505 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.504+0000 7fd167190640 1 -- 172.21.15.6:0/2103154527 >> 172.21.15.6:0/2103154527 conn(0x7fd1600617a0 msgr2=0x7fd160061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:31.505 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.504+0000 7fd167190640 1 -- 172.21.15.6:0/2103154527 shutdown_connections 2024-03-25T19:23:31.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.504+0000 7fd167190640 1 -- 172.21.15.6:0/2103154527 wait complete. 2024-03-25T19:23:31.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.505+0000 7fd167190640 10 monclient: build_initial_monmap 2024-03-25T19:23:31.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.505+0000 7fd167190640 10 monclient: monmap: 2024-03-25T19:23:31.506 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:23:31.506 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:23:31.506 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:23:31.506 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:23:31.506 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:31.506 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:23:31.506 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:23:31.506 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:23:31.506 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:23:31.506 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:31.507 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.505+0000 7fd167190640 1 librados: starting msgr at 2024-03-25T19:23:31.507 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.505+0000 7fd167190640 1 librados: starting objecter 2024-03-25T19:23:31.507 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.505+0000 7fd167190640 1 Processor -- start 2024-03-25T19:23:31.507 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.505+0000 7fd167190640 1 -- start start 2024-03-25T19:23:31.507 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.505+0000 7fd167190640 1 librados: setting wanted keys 2024-03-25T19:23:31.507 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.505+0000 7fd167190640 1 librados: calling monclient init 2024-03-25T19:23:31.507 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.505+0000 7fd167190640 10 monclient: init 2024-03-25T19:23:31.508 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.506+0000 7fd167190640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:31.508 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.506+0000 7fd167190640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:23:31.508 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.506+0000 7fd167190640 10 monclient(hunting): picked mon.noname-a con 0x7fd1600b1b00 addr v1:172.21.15.6:6789/0 2024-03-25T19:23:31.508 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.506+0000 7fd167190640 10 monclient(hunting): picked mon.noname-c con 0x7fd1600b44b0 addr v1:172.21.15.6:6790/0 2024-03-25T19:23:31.508 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.506+0000 7fd167190640 10 monclient(hunting): picked mon.noname-b con 0x7fd1600b4e70 addr v1:172.21.15.192:6789/0 2024-03-25T19:23:31.508 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.506+0000 7fd167190640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd16013cb00 con 0x7fd1600b1b00 2024-03-25T19:23:31.508 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.506+0000 7fd167190640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd16013cde0 con 0x7fd1600b4e70 2024-03-25T19:23:31.508 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.506+0000 7fd167190640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd16013b640 con 0x7fd1600b44b0 2024-03-25T19:23:31.508 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.506+0000 7fd167190640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:31.508 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.506+0000 7fd167190640 10 monclient(hunting): authenticate will time out at 6515.746710s 2024-03-25T19:23:31.519 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.519+0000 7fd164f2c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd1600b1b00 0x7fd16013a9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34288/0 (socket says 172.21.15.6:34288) 2024-03-25T19:23:31.519 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.519+0000 7fd164f2c640 1 -- 172.21.15.6:0/2437722490 learned_addr learned my addr 172.21.15.6:0/2437722490 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:23:31.520 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.520+0000 7fd15d7fa640 10 client.?.objecter ms_handle_connect 0x7fd1600b1b00 2024-03-25T19:23:31.520 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.520+0000 7fd15d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:31.521 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.521+0000 7fd15d7fa640 10 client.?.objecter ms_handle_connect 0x7fd1600b44b0 2024-03-25T19:23:31.521 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.521+0000 7fd15d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:31.522 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.522+0000 7fd15d7fa640 10 client.?.objecter ms_handle_connect 0x7fd1600b4e70 2024-03-25T19:23:31.522 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.522+0000 7fd15d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:31.528 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.528+0000 7fd15d7fa640 1 -- 172.21.15.6:0/2437722490 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1788374532 0 0) 0x7fd16013b640 con 0x7fd1600b44b0 2024-03-25T19:23:31.528 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.528+0000 7fd15d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:23:31.528 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.528+0000 7fd15d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:31.528 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.528+0000 7fd15d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:31.528 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.528+0000 7fd15d7fa640 10 monclient(hunting): my global_id is 4649 2024-03-25T19:23:31.529 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.528+0000 7fd15d7fa640 1 -- 172.21.15.6:0/2437722490 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd144004eb0 con 0x7fd1600b44b0 2024-03-25T19:23:31.536 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.536+0000 7fd15d7fa640 1 -- 172.21.15.6:0/2437722490 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3528646223 0 0) 0x7fd16013cde0 con 0x7fd1600b4e70 2024-03-25T19:23:31.536 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.536+0000 7fd15d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:23:31.536 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.536+0000 7fd15d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:31.536 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.536+0000 7fd15d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:31.536 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.536+0000 7fd15d7fa640 10 monclient(hunting): my global_id is 4657 2024-03-25T19:23:31.536 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.536+0000 7fd15d7fa640 1 -- 172.21.15.6:0/2437722490 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd16013b640 con 0x7fd1600b4e70 2024-03-25T19:23:31.540 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.540+0000 7fd15d7fa640 1 -- 172.21.15.6:0/2437722490 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2868838252 0 0) 0x7fd144004eb0 con 0x7fd1600b44b0 2024-03-25T19:23:31.540 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.540+0000 7fd15d7fa640 1 -- 172.21.15.6:0/2437722490 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd16013cde0 con 0x7fd1600b44b0 2024-03-25T19:23:31.540 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.540+0000 7fd15d7fa640 1 -- 172.21.15.6:0/2437722490 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fd14c0018c0 con 0x7fd1600b44b0 2024-03-25T19:23:31.540 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.540+0000 7fd15d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:31.540 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.540+0000 7fd15d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:23:31.541 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.540+0000 7fd15d7fa640 10 monclient(hunting): dump: 2024-03-25T19:23:31.541 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:23:31.541 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:31.541 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:31.541 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:31.541 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:31.541 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:31.541 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:23:31.541 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:23:31.541 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:23:31.541 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:31.541 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.540+0000 7fd15d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:23:31.542 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.542+0000 7fd15d7fa640 1 -- 172.21.15.6:0/2437722490 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3375127844 0 0) 0x7fd16013cb00 con 0x7fd1600b1b00 2024-03-25T19:23:31.542 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.542+0000 7fd15d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:23:31.542 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.542+0000 7fd15d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:31.542 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.542+0000 7fd15d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:31.542 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.542+0000 7fd15d7fa640 10 monclient(hunting): my global_id is 4647 2024-03-25T19:23:31.542 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.542+0000 7fd15d7fa640 1 -- 172.21.15.6:0/2437722490 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd144004eb0 con 0x7fd1600b1b00 2024-03-25T19:23:31.550 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.550+0000 7fd15d7fa640 1 -- 172.21.15.6:0/2437722490 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3764291796 0 0) 0x7fd16013cde0 con 0x7fd1600b44b0 2024-03-25T19:23:31.550 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.550+0000 7fd15d7fa640 1 -- 172.21.15.6:0/2437722490 >> v1:172.21.15.192:6789/0 conn(0x7fd1600b4e70 legacy=0x7fd160142650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:31.550 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.550+0000 7fd15d7fa640 1 -- 172.21.15.6:0/2437722490 >> v1:172.21.15.6:6789/0 conn(0x7fd1600b1b00 legacy=0x7fd16013a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:31.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.550+0000 7fd15d7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:23:31.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.550+0000 7fd15d7fa640 1 monclient: found mon.c 2024-03-25T19:23:31.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.550+0000 7fd15d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:31.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.550+0000 7fd15d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:23:31.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.550+0000 7fd15d7fa640 1 -- 172.21.15.6:0/2437722490 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd16013b1b0 con 0x7fd1600b44b0 2024-03-25T19:23:31.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.550+0000 7fd15d7fa640 10 monclient: _finish_auth 0 2024-03-25T19:23:31.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.550+0000 7fd15d7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:23:31.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.550+0000 7fd15d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:31.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.550+0000 7fd167190640 5 monclient: authenticate success, global_id 4649 2024-03-25T19:23:31.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.550+0000 7fd167190640 10 monclient: _renew_subs 2024-03-25T19:23:31.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.550+0000 7fd167190640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:23:31.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.550+0000 7fd167190640 1 -- 172.21.15.6:0/2437722490 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1601447b0 con 0x7fd1600b44b0 2024-03-25T19:23:31.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.550+0000 7fd167190640 10 client.4649.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:23:31.551 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.550+0000 7fd167190640 10 monclient: _renew_subs 2024-03-25T19:23:31.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.550+0000 7fd167190640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:23:31.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.550+0000 7fd167190640 1 -- 172.21.15.6:0/2437722490 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd16013bd40 con 0x7fd1600b44b0 2024-03-25T19:23:31.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.550+0000 7fd167190640 1 librados: init done 2024-03-25T19:23:31.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.552+0000 7fd167190640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:31.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.552+0000 7fd167190640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:31.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.552+0000 7fd167190640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:23:31.553 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.552+0000 7fd167190640 1 -- 172.21.15.6:0/2437722490 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd16013c940 con 0x7fd1600b44b0 2024-03-25T19:23:31.884 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.884+0000 7fd15d7fa640 1 -- 172.21.15.6:0/2437722490 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd14c0035f0 con 0x7fd1600b44b0 2024-03-25T19:23:31.884 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.884+0000 7fd15d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:31.884 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.884+0000 7fd15d7fa640 1 -- 172.21.15.6:0/2437722490 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fd14c003980 con 0x7fd1600b44b0 2024-03-25T19:23:31.884 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.884+0000 7fd15d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:31.884 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.884+0000 7fd15d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:23:31.884 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.884+0000 7fd15d7fa640 10 monclient: dump: 2024-03-25T19:23:31.884 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:23:31.884 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:31.884 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:31.884 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:31.884 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:31.885 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:31.885 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:23:31.885 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:23:31.885 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:23:31.885 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:31.885 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.884+0000 7fd15effd640 4 set_mon_vals no callback set 2024-03-25T19:23:31.886 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.885+0000 7fd15d7fa640 1 -- 172.21.15.6:0/2437722490 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7fd14c003b60 con 0x7fd1600b44b0 2024-03-25T19:23:31.886 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.886+0000 7fd164f2c640 10 monclient: get_auth_request con 0x7fd14406de00 auth_method 0 2024-03-25T19:23:31.886 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.886+0000 7fd15d7fa640 10 client.4649.objecter ms_handle_connect 0x7fd14406de00 2024-03-25T19:23:31.891 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.891+0000 7fd15d7fa640 1 -- 172.21.15.6:0/2437722490 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7fd14c0535d0 con 0x7fd1600b44b0 2024-03-25T19:23:31.891 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.891+0000 7fd15d7fa640 10 client.4649.objecter ms_dispatch 0x7fd160000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:23:31.891 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.891+0000 7fd15d7fa640 3 client.4649.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:23:31.891 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.891+0000 7fd15d7fa640 3 client.4649.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:23:31.891 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.891+0000 7fd15d7fa640 20 client.4649.objecter dump_active .. 0 homeless 2024-03-25T19:23:31.891 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.891+0000 7fd15d7fa640 1 -- 172.21.15.6:0/2437722490 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd14c04f0e0 con 0x7fd1600b44b0 2024-03-25T19:23:31.891 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.891+0000 7fd15d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:31.891 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:31.891+0000 7fd15d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:23:32.111 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:32.112+0000 7fd167190640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:32.112 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:32.112+0000 7fd167190640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:32.112 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:32.112+0000 7fd167190640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:23:32.112 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:32.112+0000 7fd167190640 1 -- 172.21.15.6:0/2437722490 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd16013b7d0 con 0x7fd1600b44b0 2024-03-25T19:23:32.137 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:32.137+0000 7fd15d7fa640 1 -- 172.21.15.6:0/2437722490 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7fd14c051010 con 0x7fd1600b44b0 2024-03-25T19:23:32.137 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:32.137+0000 7fd15d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:32.137 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:32.137+0000 7fd15d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:23:32.137 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:23:32.137 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:23:32.139 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:32.139+0000 7fd167190640 10 librados: watch_flush enter 2024-03-25T19:23:32.139 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:32.139+0000 7fd167190640 10 librados: watch_flush exit 2024-03-25T19:23:32.140 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:32.139+0000 7fd167190640 20 client.4649.objecter shutdown clearing up homeless session... 2024-03-25T19:23:32.140 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:32.139+0000 7fd167190640 10 client.4649.objecter successfully canceled tick 2024-03-25T19:23:32.140 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:32.139+0000 7fd167190640 1 -- 172.21.15.6:0/2437722490 >> v1:172.21.15.6:6800/2936652650 conn(0x7fd14406de00 legacy=0x7fd1440702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:32.140 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:32.139+0000 7fd167190640 10 monclient: shutdown 2024-03-25T19:23:32.140 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:32.139+0000 7fd167190640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:32.140 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:32.139+0000 7fd167190640 1 -- 172.21.15.6:0/2437722490 >> v1:172.21.15.6:6790/0 conn(0x7fd1600b44b0 legacy=0x7fd1601440a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:32.140 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:32.139+0000 7fd167190640 1 -- 172.21.15.6:0/2437722490 shutdown_connections 2024-03-25T19:23:32.140 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:32.139+0000 7fd167190640 1 -- 172.21.15.6:0/2437722490 >> 172.21.15.6:0/2437722490 conn(0x7fd1600617a0 msgr2=0x7fd1600b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:32.140 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:32.139+0000 7fd167190640 1 -- 172.21.15.6:0/2437722490 shutdown_connections 2024-03-25T19:23:32.140 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:32.139+0000 7fd167190640 1 -- 172.21.15.6:0/2437722490 wait complete. 2024-03-25T19:23:32.140 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:32.139+0000 7fd167190640 1 librados: shutdown 2024-03-25T19:23:32.151 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:23:38.958 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:23:39.073 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.073+0000 7fb2b017b640 10 monclient: get_monmap_and_config 2024-03-25T19:23:39.073 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.073+0000 7fb2b017b640 10 monclient: build_initial_monmap 2024-03-25T19:23:39.074 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.073+0000 7fb2b017b640 10 monclient: monmap: 2024-03-25T19:23:39.074 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:23:39.074 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:39.074 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:23:39.074754+0000 2024-03-25T19:23:39.074 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:23:39.074754+0000 2024-03-25T19:23:39.074 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:39.074 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:39.074 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:23:39.074 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:23:39.074 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:23:39.074 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:39.076 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.074+0000 7fb2b017b640 1 Processor -- start 2024-03-25T19:23:39.076 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.074+0000 7fb2b017b640 1 -- start start 2024-03-25T19:23:39.076 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.074+0000 7fb2b017b640 10 monclient: init 2024-03-25T19:23:39.076 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.075+0000 7fb2b017b640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:39.076 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.075+0000 7fb2b017b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:23:39.076 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.075+0000 7fb2b017b640 10 monclient(hunting): picked mon.noname-c con 0x7fb2a80b4ff0 addr v1:172.21.15.192:6789/0 2024-03-25T19:23:39.076 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.075+0000 7fb2b017b640 10 monclient(hunting): picked mon.noname-b con 0x7fb2a80b4670 addr v1:172.21.15.6:6790/0 2024-03-25T19:23:39.076 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.075+0000 7fb2b017b640 10 monclient(hunting): picked mon.noname-a con 0x7fb2a813c480 addr v1:172.21.15.6:6789/0 2024-03-25T19:23:39.076 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.075+0000 7fb2b017b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2a80ab0e0 con 0x7fb2a813c480 2024-03-25T19:23:39.076 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.075+0000 7fb2b017b640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2a80ab3c0 con 0x7fb2a80b4ff0 2024-03-25T19:23:39.076 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.075+0000 7fb2b017b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2a80a9f30 con 0x7fb2a80b4670 2024-03-25T19:23:39.076 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.075+0000 7fb2b017b640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:39.076 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.075+0000 7fb2b017b640 10 monclient(hunting): authenticate will time out at 6523.315345s 2024-03-25T19:23:39.085 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.086+0000 7fb2adf17640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fb2a80b4ff0 0x7fb2a813bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:45244/0 (socket says 172.21.15.6:45244) 2024-03-25T19:23:39.085 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.086+0000 7fb2adf17640 1 -- 172.21.15.6:0/3568560442 learned_addr learned my addr 172.21.15.6:0/3568560442 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:23:39.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.099+0000 7fb2acf15640 1 -- 172.21.15.6:0/3568560442 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2147953147 0 0) 0x7fb2a80ab3c0 con 0x7fb2a80b4ff0 2024-03-25T19:23:39.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.099+0000 7fb2acf15640 10 monclient(hunting): _negotiate 2024-03-25T19:23:39.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.099+0000 7fb2acf15640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:39.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.099+0000 7fb2acf15640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:39.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.099+0000 7fb2acf15640 10 monclient(hunting): my global_id is 4660 2024-03-25T19:23:39.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.099+0000 7fb2acf15640 1 -- 172.21.15.6:0/3568560442 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb294003520 con 0x7fb2a80b4ff0 2024-03-25T19:23:39.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.105+0000 7fb2acf15640 1 -- 172.21.15.6:0/3568560442 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1991710100 0 0) 0x7fb2a80ab0e0 con 0x7fb2a813c480 2024-03-25T19:23:39.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.105+0000 7fb2acf15640 10 monclient(hunting): _negotiate 2024-03-25T19:23:39.105 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.105+0000 7fb2acf15640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:39.105 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.105+0000 7fb2acf15640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:39.105 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.105+0000 7fb2acf15640 10 monclient(hunting): my global_id is 4650 2024-03-25T19:23:39.105 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.105+0000 7fb2acf15640 1 -- 172.21.15.6:0/3568560442 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2a80ab3c0 con 0x7fb2a813c480 2024-03-25T19:23:39.108 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.108+0000 7fb2acf15640 1 -- 172.21.15.6:0/3568560442 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1078021352 0 0) 0x7fb2a80a9f30 con 0x7fb2a80b4670 2024-03-25T19:23:39.108 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.108+0000 7fb2acf15640 10 monclient(hunting): _negotiate 2024-03-25T19:23:39.108 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.108+0000 7fb2acf15640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:39.108 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.108+0000 7fb2acf15640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:39.108 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.108+0000 7fb2acf15640 10 monclient(hunting): my global_id is 4652 2024-03-25T19:23:39.108 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.108+0000 7fb2acf15640 1 -- 172.21.15.6:0/3568560442 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2a80ab0e0 con 0x7fb2a80b4670 2024-03-25T19:23:39.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.116+0000 7fb2acf15640 1 -- 172.21.15.6:0/3568560442 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 729520044 0 0) 0x7fb2a80ab3c0 con 0x7fb2a813c480 2024-03-25T19:23:39.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.116+0000 7fb2acf15640 1 -- 172.21.15.6:0/3568560442 >> v1:172.21.15.6:6790/0 conn(0x7fb2a80b4670 legacy=0x7fb2a81425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:39.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.116+0000 7fb2acf15640 1 -- 172.21.15.6:0/3568560442 >> v1:172.21.15.192:6789/0 conn(0x7fb2a80b4ff0 legacy=0x7fb2a813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:39.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.116+0000 7fb2acf15640 10 monclient: _finish_hunting 0 2024-03-25T19:23:39.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.116+0000 7fb2acf15640 1 monclient: found mon.noname-a 2024-03-25T19:23:39.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.116+0000 7fb2acf15640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:39.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.116+0000 7fb2acf15640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-03-25T19:23:39.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.116+0000 7fb2acf15640 1 -- 172.21.15.6:0/3568560442 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2a80a9b70 con 0x7fb2a813c480 2024-03-25T19:23:39.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.116+0000 7fb2acf15640 10 monclient: _finish_auth 0 2024-03-25T19:23:39.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.116+0000 7fb2acf15640 10 monclient: _check_auth_tickets 2024-03-25T19:23:39.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.116+0000 7fb2acf15640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:39.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.116+0000 7fb2acf15640 1 -- 172.21.15.6:0/3568560442 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fb2a00019f0 con 0x7fb2a813c480 2024-03-25T19:23:39.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.116+0000 7fb2acf15640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:39.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.116+0000 7fb2acf15640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:23:39.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.116+0000 7fb2acf15640 10 monclient: dump: 2024-03-25T19:23:39.118 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:23:39.118 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:39.118 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:39.118 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:39.118 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:39.118 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:39.118 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:23:39.118 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:23:39.118 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:23:39.118 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:39.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.116+0000 7fb2b017b640 5 monclient: authenticate success, global_id 4650 2024-03-25T19:23:39.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.116+0000 7fb2b017b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:23:39.132 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.132+0000 7fb2acf15640 1 -- 172.21.15.6:0/3568560442 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2a0001b50 con 0x7fb2a813c480 2024-03-25T19:23:39.132 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.132+0000 7fb2acf15640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:39.132 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.132+0000 7fb2acf15640 1 -- 172.21.15.6:0/3568560442 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fb2a0001f60 con 0x7fb2a813c480 2024-03-25T19:23:39.132 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.132+0000 7fb2b017b640 10 monclient: get_monmap_and_config success 2024-03-25T19:23:39.132 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.132+0000 7fb2b017b640 4 set_mon_vals no callback set 2024-03-25T19:23:39.133 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.132+0000 7fb2acf15640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:39.133 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.132+0000 7fb2acf15640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:23:39.133 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.132+0000 7fb2acf15640 10 monclient: dump: 2024-03-25T19:23:39.133 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:23:39.133 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:39.133 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:39.133 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:39.133 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:39.133 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:39.133 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:23:39.133 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:23:39.133 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:23:39.133 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:39.135 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.133+0000 7fb2b017b640 10 monclient: shutdown 2024-03-25T19:23:39.135 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.133+0000 7fb2b017b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:39.136 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.133+0000 7fb2b017b640 1 -- 172.21.15.6:0/3568560442 >> v1:172.21.15.6:6789/0 conn(0x7fb2a813c480 legacy=0x7fb2a814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:39.136 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.133+0000 7fb2b017b640 1 -- 172.21.15.6:0/3568560442 shutdown_connections 2024-03-25T19:23:39.136 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.133+0000 7fb2b017b640 1 -- 172.21.15.6:0/3568560442 >> 172.21.15.6:0/3568560442 conn(0x7fb2a80617a0 msgr2=0x7fb2a8061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:39.136 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.133+0000 7fb2b017b640 1 -- 172.21.15.6:0/3568560442 shutdown_connections 2024-03-25T19:23:39.136 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.133+0000 7fb2b017b640 1 -- 172.21.15.6:0/3568560442 wait complete. 2024-03-25T19:23:39.136 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.134+0000 7fb2b017b640 10 monclient: build_initial_monmap 2024-03-25T19:23:39.136 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.134+0000 7fb2b017b640 10 monclient: monmap: 2024-03-25T19:23:39.136 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:23:39.136 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:23:39.136 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:23:39.136 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:23:39.136 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:39.136 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:23:39.136 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:23:39.137 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:23:39.137 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:23:39.137 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:39.137 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.134+0000 7fb2b017b640 1 librados: starting msgr at 2024-03-25T19:23:39.137 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.134+0000 7fb2b017b640 1 librados: starting objecter 2024-03-25T19:23:39.137 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.135+0000 7fb2b017b640 1 Processor -- start 2024-03-25T19:23:39.137 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.135+0000 7fb2b017b640 1 -- start start 2024-03-25T19:23:39.137 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.135+0000 7fb2b017b640 1 librados: setting wanted keys 2024-03-25T19:23:39.137 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.135+0000 7fb2b017b640 1 librados: calling monclient init 2024-03-25T19:23:39.137 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.135+0000 7fb2b017b640 10 monclient: init 2024-03-25T19:23:39.137 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.135+0000 7fb2b017b640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:39.137 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.135+0000 7fb2b017b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:23:39.137 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.135+0000 7fb2b017b640 10 monclient(hunting): picked mon.noname-c con 0x7fb2a80b4670 addr v1:172.21.15.6:6790/0 2024-03-25T19:23:39.137 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.135+0000 7fb2b017b640 10 monclient(hunting): picked mon.noname-a con 0x7fb2a80b4ff0 addr v1:172.21.15.6:6789/0 2024-03-25T19:23:39.137 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.135+0000 7fb2b017b640 10 monclient(hunting): picked mon.noname-b con 0x7fb2a813c480 addr v1:172.21.15.192:6789/0 2024-03-25T19:23:39.138 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.135+0000 7fb2b017b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2a81446b0 con 0x7fb2a80b4ff0 2024-03-25T19:23:39.138 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.135+0000 7fb2b017b640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2a81462f0 con 0x7fb2a813c480 2024-03-25T19:23:39.138 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.135+0000 7fb2b017b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2a8144900 con 0x7fb2a80b4670 2024-03-25T19:23:39.138 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.135+0000 7fb2b017b640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:39.138 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.135+0000 7fb2b017b640 10 monclient(hunting): authenticate will time out at 6523.375525s 2024-03-25T19:23:39.146 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.146+0000 7fb2adf17640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fb2a80b4670 0x7fb2a81437f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:41202/0 (socket says 172.21.15.6:41202) 2024-03-25T19:23:39.146 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.146+0000 7fb2adf17640 1 -- 172.21.15.6:0/1036367709 learned_addr learned my addr 172.21.15.6:0/1036367709 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:23:39.146 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.147+0000 7fb29e7fc640 10 client.?.objecter ms_handle_connect 0x7fb2a80b4670 2024-03-25T19:23:39.146 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.147+0000 7fb29e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:39.150 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.151+0000 7fb29e7fc640 10 client.?.objecter ms_handle_connect 0x7fb2a813c480 2024-03-25T19:23:39.150 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.151+0000 7fb29e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:39.154 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.154+0000 7fb29e7fc640 1 -- 172.21.15.6:0/1036367709 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2767535005 0 0) 0x7fb2a81462f0 con 0x7fb2a813c480 2024-03-25T19:23:39.154 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.154+0000 7fb29e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:23:39.154 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.154+0000 7fb29e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:39.154 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.154+0000 7fb29e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:39.155 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.154+0000 7fb29e7fc640 10 monclient(hunting): my global_id is 4663 2024-03-25T19:23:39.155 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.154+0000 7fb29e7fc640 1 -- 172.21.15.6:0/1036367709 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb28c004eb0 con 0x7fb2a813c480 2024-03-25T19:23:39.157 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.157+0000 7fb29e7fc640 1 -- 172.21.15.6:0/1036367709 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 644101062 0 0) 0x7fb2a8144900 con 0x7fb2a80b4670 2024-03-25T19:23:39.157 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.157+0000 7fb29e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:23:39.157 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.157+0000 7fb29e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:39.157 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.157+0000 7fb29e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:39.157 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.157+0000 7fb29e7fc640 10 monclient(hunting): my global_id is 4655 2024-03-25T19:23:39.157 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.157+0000 7fb29e7fc640 1 -- 172.21.15.6:0/1036367709 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2a81462f0 con 0x7fb2a80b4670 2024-03-25T19:23:39.164 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.164+0000 7fb29e7fc640 10 client.?.objecter ms_handle_connect 0x7fb2a80b4ff0 2024-03-25T19:23:39.164 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.164+0000 7fb29e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:39.166 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.166+0000 7fb29e7fc640 1 -- 172.21.15.6:0/1036367709 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3280393520 0 0) 0x7fb28c004eb0 con 0x7fb2a813c480 2024-03-25T19:23:39.166 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.166+0000 7fb29e7fc640 1 -- 172.21.15.6:0/1036367709 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2a8144900 con 0x7fb2a813c480 2024-03-25T19:23:39.167 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.166+0000 7fb29e7fc640 1 -- 172.21.15.6:0/1036367709 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fb2a0001ea0 con 0x7fb2a813c480 2024-03-25T19:23:39.167 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.166+0000 7fb29e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:39.167 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.166+0000 7fb29e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:23:39.167 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.166+0000 7fb29e7fc640 10 monclient(hunting): dump: 2024-03-25T19:23:39.167 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:23:39.167 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:39.167 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:39.167 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:39.167 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:39.167 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:39.167 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:23:39.167 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:23:39.167 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:23:39.167 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:39.168 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.166+0000 7fb29e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:23:39.178 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.178+0000 7fb29e7fc640 1 -- 172.21.15.6:0/1036367709 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2282169047 0 0) 0x7fb2a8144900 con 0x7fb2a813c480 2024-03-25T19:23:39.179 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.178+0000 7fb29e7fc640 1 -- 172.21.15.6:0/1036367709 >> v1:172.21.15.6:6790/0 conn(0x7fb2a80b4670 legacy=0x7fb2a81437f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:39.179 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.178+0000 7fb29e7fc640 1 -- 172.21.15.6:0/1036367709 >> v1:172.21.15.6:6789/0 conn(0x7fb2a80b4ff0 legacy=0x7fb2a813c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:39.179 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.178+0000 7fb29e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:23:39.179 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.178+0000 7fb29e7fc640 1 monclient: found mon.b 2024-03-25T19:23:39.179 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.178+0000 7fb29e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:39.179 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.178+0000 7fb29e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:23:39.179 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.178+0000 7fb29e7fc640 1 -- 172.21.15.6:0/1036367709 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2a8144be0 con 0x7fb2a813c480 2024-03-25T19:23:39.179 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.178+0000 7fb29e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:23:39.179 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.178+0000 7fb29e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:23:39.179 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.178+0000 7fb29e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:39.179 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.178+0000 7fb2b017b640 5 monclient: authenticate success, global_id 4663 2024-03-25T19:23:39.179 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.178+0000 7fb2b017b640 10 monclient: _renew_subs 2024-03-25T19:23:39.179 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.178+0000 7fb2b017b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:23:39.179 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.178+0000 7fb2b017b640 1 -- 172.21.15.6:0/1036367709 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2a8144ec0 con 0x7fb2a813c480 2024-03-25T19:23:39.180 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.178+0000 7fb2b017b640 10 client.4663.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:23:39.180 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.178+0000 7fb2b017b640 10 monclient: _renew_subs 2024-03-25T19:23:39.180 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.178+0000 7fb2b017b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:23:39.180 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.178+0000 7fb2b017b640 1 -- 172.21.15.6:0/1036367709 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2a813d220 con 0x7fb2a813c480 2024-03-25T19:23:39.180 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.178+0000 7fb2b017b640 1 librados: init done 2024-03-25T19:23:39.180 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.179+0000 7fb2b017b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:39.180 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.179+0000 7fb2b017b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:39.181 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.179+0000 7fb2b017b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:23:39.181 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.179+0000 7fb2b017b640 1 -- 172.21.15.6:0/1036367709 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2a813ee50 con 0x7fb2a813c480 2024-03-25T19:23:39.456 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.456+0000 7fb29e7fc640 1 -- 172.21.15.6:0/1036367709 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2a00020c0 con 0x7fb2a813c480 2024-03-25T19:23:39.456 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.456+0000 7fb29e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:39.456 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.456+0000 7fb29e7fc640 1 -- 172.21.15.6:0/1036367709 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fb2a0003c20 con 0x7fb2a813c480 2024-03-25T19:23:39.456 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.456+0000 7fb29e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:39.456 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.456+0000 7fb29e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:23:39.456 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.456+0000 7fb29e7fc640 10 monclient: dump: 2024-03-25T19:23:39.456 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:23:39.457 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:39.457 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:39.457 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:39.457 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:39.457 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:39.457 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:23:39.457 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:23:39.457 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:23:39.457 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:39.457 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.456+0000 7fb29ffff640 4 set_mon_vals no callback set 2024-03-25T19:23:39.458 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.457+0000 7fb29e7fc640 1 -- 172.21.15.6:0/1036367709 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7fb2a0019bf0 con 0x7fb2a813c480 2024-03-25T19:23:39.458 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.457+0000 7fb2adf17640 10 monclient: get_auth_request con 0x7fb28c06dda0 auth_method 0 2024-03-25T19:23:39.458 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.458+0000 7fb29e7fc640 10 client.4663.objecter ms_handle_connect 0x7fb28c06dda0 2024-03-25T19:23:39.461 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.461+0000 7fb29e7fc640 1 -- 172.21.15.6:0/1036367709 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7fb2a00537c0 con 0x7fb2a813c480 2024-03-25T19:23:39.461 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.461+0000 7fb29e7fc640 10 client.4663.objecter ms_dispatch 0x7fb2a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:23:39.461 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.461+0000 7fb29e7fc640 3 client.4663.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:23:39.461 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.461+0000 7fb29e7fc640 3 client.4663.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:23:39.461 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.461+0000 7fb29e7fc640 20 client.4663.objecter dump_active .. 0 homeless 2024-03-25T19:23:39.461 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.462+0000 7fb29e7fc640 1 -- 172.21.15.6:0/1036367709 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb2a004f100 con 0x7fb2a813c480 2024-03-25T19:23:39.462 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.462+0000 7fb29e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:39.462 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.462+0000 7fb29e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:23:39.689 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.689+0000 7fb2b017b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:39.689 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.689+0000 7fb2b017b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:39.690 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.689+0000 7fb2b017b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:23:39.690 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.689+0000 7fb2b017b640 1 -- 172.21.15.6:0/1036367709 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2a8146790 con 0x7fb2a813c480 2024-03-25T19:23:39.716 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.716+0000 7fb29e7fc640 1 -- 172.21.15.6:0/1036367709 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7fb2a0051200 con 0x7fb2a813c480 2024-03-25T19:23:39.716 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.716+0000 7fb29e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:39.716 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.716+0000 7fb29e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:23:39.716 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:23:39.716 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:23:39.719 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.718+0000 7fb2b017b640 10 librados: watch_flush enter 2024-03-25T19:23:39.719 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.718+0000 7fb2b017b640 10 librados: watch_flush exit 2024-03-25T19:23:39.719 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.718+0000 7fb2b017b640 20 client.4663.objecter shutdown clearing up homeless session... 2024-03-25T19:23:39.719 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.718+0000 7fb2b017b640 10 client.4663.objecter successfully canceled tick 2024-03-25T19:23:39.719 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.718+0000 7fb2b017b640 1 -- 172.21.15.6:0/1036367709 >> v1:172.21.15.6:6800/2936652650 conn(0x7fb28c06dda0 legacy=0x7fb28c070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:39.719 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.718+0000 7fb2b017b640 10 monclient: shutdown 2024-03-25T19:23:39.719 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.718+0000 7fb2b017b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:39.719 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.718+0000 7fb2b017b640 1 -- 172.21.15.6:0/1036367709 >> v1:172.21.15.192:6789/0 conn(0x7fb2a813c480 legacy=0x7fb2a81e2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:39.719 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.718+0000 7fb2b017b640 1 -- 172.21.15.6:0/1036367709 shutdown_connections 2024-03-25T19:23:39.719 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.718+0000 7fb2b017b640 1 -- 172.21.15.6:0/1036367709 >> 172.21.15.6:0/1036367709 conn(0x7fb2a80617a0 msgr2=0x7fb2a80b32f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:39.719 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.718+0000 7fb2b017b640 1 -- 172.21.15.6:0/1036367709 shutdown_connections 2024-03-25T19:23:39.720 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.718+0000 7fb2b017b640 1 -- 172.21.15.6:0/1036367709 wait complete. 2024-03-25T19:23:39.720 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:39.718+0000 7fb2b017b640 1 librados: shutdown 2024-03-25T19:23:39.729 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:23:46.536 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:23:46.652 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.652+0000 7eff78177640 10 monclient: get_monmap_and_config 2024-03-25T19:23:46.652 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.652+0000 7eff78177640 10 monclient: build_initial_monmap 2024-03-25T19:23:46.652 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.652+0000 7eff78177640 10 monclient: monmap: 2024-03-25T19:23:46.652 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:23:46.652 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:46.652 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:23:46.653093+0000 2024-03-25T19:23:46.652 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:23:46.653093+0000 2024-03-25T19:23:46.652 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:46.652 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:46.652 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:23:46.653 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:23:46.653 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:23:46.653 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:46.654 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.652+0000 7eff78177640 1 Processor -- start 2024-03-25T19:23:46.654 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.653+0000 7eff78177640 1 -- start start 2024-03-25T19:23:46.654 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.653+0000 7eff78177640 10 monclient: init 2024-03-25T19:23:46.654 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.653+0000 7eff78177640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:46.654 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.653+0000 7eff78177640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:23:46.654 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.653+0000 7eff78177640 10 monclient(hunting): picked mon.noname-c con 0x7eff700b4e70 addr v1:172.21.15.192:6789/0 2024-03-25T19:23:46.654 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.653+0000 7eff78177640 10 monclient(hunting): picked mon.noname-b con 0x7eff700b44b0 addr v1:172.21.15.6:6790/0 2024-03-25T19:23:46.654 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.653+0000 7eff78177640 10 monclient(hunting): picked mon.noname-a con 0x7eff700b1b00 addr v1:172.21.15.6:6789/0 2024-03-25T19:23:46.655 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.653+0000 7eff78177640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff700a96f0 con 0x7eff700b1b00 2024-03-25T19:23:46.655 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.654+0000 7eff78177640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff700ab0e0 con 0x7eff700b4e70 2024-03-25T19:23:46.655 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.654+0000 7eff78177640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff70145f70 con 0x7eff700b44b0 2024-03-25T19:23:46.655 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.654+0000 7eff78177640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:46.655 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.654+0000 7eff78177640 10 monclient(hunting): authenticate will time out at 6530.893817s 2024-03-25T19:23:46.665 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.664+0000 7eff75f13640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7eff700b4e70 0x7eff7013f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:49884/0 (socket says 172.21.15.6:49884) 2024-03-25T19:23:46.665 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.664+0000 7eff75f13640 1 -- 172.21.15.6:0/1422801483 learned_addr learned my addr 172.21.15.6:0/1422801483 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:23:46.681 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.680+0000 7eff74f11640 1 -- 172.21.15.6:0/1422801483 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2156785757 0 0) 0x7eff700ab0e0 con 0x7eff700b4e70 2024-03-25T19:23:46.681 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.680+0000 7eff74f11640 10 monclient(hunting): _negotiate 2024-03-25T19:23:46.681 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.680+0000 7eff74f11640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:46.681 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.680+0000 7eff74f11640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:46.681 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.680+0000 7eff74f11640 10 monclient(hunting): my global_id is 4666 2024-03-25T19:23:46.681 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.680+0000 7eff74f11640 1 -- 172.21.15.6:0/1422801483 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff5c003520 con 0x7eff700b4e70 2024-03-25T19:23:46.684 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.683+0000 7eff74f11640 1 -- 172.21.15.6:0/1422801483 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3394351020 0 0) 0x7eff70145f70 con 0x7eff700b44b0 2024-03-25T19:23:46.684 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.683+0000 7eff74f11640 10 monclient(hunting): _negotiate 2024-03-25T19:23:46.684 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.683+0000 7eff74f11640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:46.684 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.683+0000 7eff74f11640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:46.684 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.683+0000 7eff74f11640 10 monclient(hunting): my global_id is 4658 2024-03-25T19:23:46.684 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.683+0000 7eff74f11640 1 -- 172.21.15.6:0/1422801483 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff700ab0e0 con 0x7eff700b44b0 2024-03-25T19:23:46.684 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.683+0000 7eff74f11640 1 -- 172.21.15.6:0/1422801483 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4265261092 0 0) 0x7eff700a96f0 con 0x7eff700b1b00 2024-03-25T19:23:46.684 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.683+0000 7eff74f11640 10 monclient(hunting): _negotiate 2024-03-25T19:23:46.684 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.683+0000 7eff74f11640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:46.684 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.683+0000 7eff74f11640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:46.684 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.683+0000 7eff74f11640 10 monclient(hunting): my global_id is 4656 2024-03-25T19:23:46.684 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.683+0000 7eff74f11640 1 -- 172.21.15.6:0/1422801483 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff70145f70 con 0x7eff700b1b00 2024-03-25T19:23:46.693 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.693+0000 7eff74f11640 1 -- 172.21.15.6:0/1422801483 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1229268674 0 0) 0x7eff700ab0e0 con 0x7eff700b44b0 2024-03-25T19:23:46.693 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.693+0000 7eff74f11640 1 -- 172.21.15.6:0/1422801483 >> v1:172.21.15.192:6789/0 conn(0x7eff700b4e70 legacy=0x7eff7013f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:46.693 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.693+0000 7eff74f11640 1 -- 172.21.15.6:0/1422801483 >> v1:172.21.15.6:6789/0 conn(0x7eff700b1b00 legacy=0x7eff7014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:46.693 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.693+0000 7eff74f11640 10 monclient: _finish_hunting 0 2024-03-25T19:23:46.693 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.693+0000 7eff74f11640 1 monclient: found mon.noname-b 2024-03-25T19:23:46.693 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.693+0000 7eff74f11640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:46.693 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.693+0000 7eff74f11640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-03-25T19:23:46.693 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.693+0000 7eff74f11640 1 -- 172.21.15.6:0/1422801483 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff700a9b80 con 0x7eff700b44b0 2024-03-25T19:23:46.694 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.693+0000 7eff74f11640 10 monclient: _finish_auth 0 2024-03-25T19:23:46.694 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.693+0000 7eff74f11640 10 monclient: _check_auth_tickets 2024-03-25T19:23:46.694 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.693+0000 7eff74f11640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:46.694 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.693+0000 7eff74f11640 1 -- 172.21.15.6:0/1422801483 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7eff60001670 con 0x7eff700b44b0 2024-03-25T19:23:46.694 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.693+0000 7eff74f11640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:46.694 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.693+0000 7eff74f11640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:23:46.694 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.693+0000 7eff74f11640 10 monclient: dump: 2024-03-25T19:23:46.694 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:23:46.694 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:46.694 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:46.694 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:46.694 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:46.694 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:46.694 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:23:46.694 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:23:46.695 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:23:46.695 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:46.695 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.693+0000 7eff78177640 5 monclient: authenticate success, global_id 4658 2024-03-25T19:23:46.695 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.693+0000 7eff78177640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:23:46.720 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.720+0000 7eff74f11640 1 -- 172.21.15.6:0/1422801483 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7eff600017d0 con 0x7eff700b44b0 2024-03-25T19:23:46.720 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.720+0000 7eff74f11640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:46.720 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.720+0000 7eff74f11640 1 -- 172.21.15.6:0/1422801483 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7eff60001be0 con 0x7eff700b44b0 2024-03-25T19:23:46.720 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.720+0000 7eff78177640 10 monclient: get_monmap_and_config success 2024-03-25T19:23:46.720 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.720+0000 7eff78177640 4 set_mon_vals no callback set 2024-03-25T19:23:46.720 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.720+0000 7eff74f11640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:46.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.720+0000 7eff74f11640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:23:46.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.720+0000 7eff74f11640 10 monclient: dump: 2024-03-25T19:23:46.721 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:23:46.721 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:46.721 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:46.721 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:46.721 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:46.721 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:46.721 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:23:46.721 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:23:46.721 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:23:46.721 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:46.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.721+0000 7eff78177640 10 monclient: shutdown 2024-03-25T19:23:46.724 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.721+0000 7eff78177640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:46.724 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.721+0000 7eff78177640 1 -- 172.21.15.6:0/1422801483 >> v1:172.21.15.6:6790/0 conn(0x7eff700b44b0 legacy=0x7eff70145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:46.724 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.721+0000 7eff78177640 1 -- 172.21.15.6:0/1422801483 shutdown_connections 2024-03-25T19:23:46.724 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.721+0000 7eff78177640 1 -- 172.21.15.6:0/1422801483 >> 172.21.15.6:0/1422801483 conn(0x7eff700617a0 msgr2=0x7eff70061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:46.724 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.721+0000 7eff78177640 1 -- 172.21.15.6:0/1422801483 shutdown_connections 2024-03-25T19:23:46.724 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.721+0000 7eff78177640 1 -- 172.21.15.6:0/1422801483 wait complete. 2024-03-25T19:23:46.724 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.722+0000 7eff78177640 10 monclient: build_initial_monmap 2024-03-25T19:23:46.724 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.722+0000 7eff78177640 10 monclient: monmap: 2024-03-25T19:23:46.724 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:23:46.724 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:23:46.724 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:23:46.724 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:23:46.724 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:46.724 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:23:46.725 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:23:46.725 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:23:46.725 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:23:46.725 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:46.725 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.722+0000 7eff78177640 1 librados: starting msgr at 2024-03-25T19:23:46.725 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.722+0000 7eff78177640 1 librados: starting objecter 2024-03-25T19:23:46.725 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.723+0000 7eff78177640 1 Processor -- start 2024-03-25T19:23:46.725 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.723+0000 7eff78177640 1 -- start start 2024-03-25T19:23:46.725 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.723+0000 7eff78177640 1 librados: setting wanted keys 2024-03-25T19:23:46.725 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.723+0000 7eff78177640 1 librados: calling monclient init 2024-03-25T19:23:46.725 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.723+0000 7eff78177640 10 monclient: init 2024-03-25T19:23:46.725 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.723+0000 7eff78177640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:46.725 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.723+0000 7eff78177640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:23:46.725 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.723+0000 7eff78177640 10 monclient(hunting): picked mon.noname-a con 0x7eff700b1b00 addr v1:172.21.15.6:6789/0 2024-03-25T19:23:46.726 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.723+0000 7eff78177640 10 monclient(hunting): picked mon.noname-c con 0x7eff700b44b0 addr v1:172.21.15.6:6790/0 2024-03-25T19:23:46.726 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.723+0000 7eff78177640 10 monclient(hunting): picked mon.noname-b con 0x7eff700b4e70 addr v1:172.21.15.192:6789/0 2024-03-25T19:23:46.726 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.723+0000 7eff78177640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff70136f90 con 0x7eff700b1b00 2024-03-25T19:23:46.726 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.723+0000 7eff78177640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff701d79d0 con 0x7eff700b4e70 2024-03-25T19:23:46.726 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.723+0000 7eff78177640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff70137e60 con 0x7eff700b44b0 2024-03-25T19:23:46.726 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.723+0000 7eff78177640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:46.726 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.723+0000 7eff78177640 10 monclient(hunting): authenticate will time out at 6530.963605s 2024-03-25T19:23:46.731 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.732+0000 7eff75f13640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7eff700b1b00 0x7eff70136880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45570/0 (socket says 172.21.15.6:45570) 2024-03-25T19:23:46.732 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.732+0000 7eff75f13640 1 -- 172.21.15.6:0/3682870313 learned_addr learned my addr 172.21.15.6:0/3682870313 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:23:46.735 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.735+0000 7eff667fc640 10 client.?.objecter ms_handle_connect 0x7eff700b1b00 2024-03-25T19:23:46.735 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.735+0000 7eff667fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:46.741 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.741+0000 7eff667fc640 10 client.?.objecter ms_handle_connect 0x7eff700b44b0 2024-03-25T19:23:46.741 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.741+0000 7eff667fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:46.743 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.743+0000 7eff667fc640 10 client.?.objecter ms_handle_connect 0x7eff700b4e70 2024-03-25T19:23:46.743 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.743+0000 7eff667fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:46.749 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.749+0000 7eff667fc640 1 -- 172.21.15.6:0/3682870313 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2506694202 0 0) 0x7eff70136f90 con 0x7eff700b1b00 2024-03-25T19:23:46.749 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.749+0000 7eff667fc640 10 monclient(hunting): _negotiate 2024-03-25T19:23:46.750 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.749+0000 7eff667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:46.750 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.749+0000 7eff667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:46.750 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.749+0000 7eff667fc640 10 monclient(hunting): my global_id is 4659 2024-03-25T19:23:46.750 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.749+0000 7eff667fc640 1 -- 172.21.15.6:0/3682870313 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff54004eb0 con 0x7eff700b1b00 2024-03-25T19:23:46.758 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.758+0000 7eff667fc640 1 -- 172.21.15.6:0/3682870313 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1767216300 0 0) 0x7eff70137e60 con 0x7eff700b44b0 2024-03-25T19:23:46.758 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.758+0000 7eff667fc640 10 monclient(hunting): _negotiate 2024-03-25T19:23:46.758 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.758+0000 7eff667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:46.758 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.758+0000 7eff667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:46.758 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.758+0000 7eff667fc640 10 monclient(hunting): my global_id is 4661 2024-03-25T19:23:46.758 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.758+0000 7eff667fc640 1 -- 172.21.15.6:0/3682870313 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff70136f90 con 0x7eff700b44b0 2024-03-25T19:23:46.761 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.761+0000 7eff667fc640 1 -- 172.21.15.6:0/3682870313 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3867216091 0 0) 0x7eff701d79d0 con 0x7eff700b4e70 2024-03-25T19:23:46.761 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.761+0000 7eff667fc640 10 monclient(hunting): _negotiate 2024-03-25T19:23:46.761 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.761+0000 7eff667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:46.761 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.761+0000 7eff667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:46.762 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.761+0000 7eff667fc640 10 monclient(hunting): my global_id is 4669 2024-03-25T19:23:46.762 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.761+0000 7eff667fc640 1 -- 172.21.15.6:0/3682870313 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff70137e60 con 0x7eff700b4e70 2024-03-25T19:23:46.770 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.770+0000 7eff667fc640 1 -- 172.21.15.6:0/3682870313 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3457010406 0 0) 0x7eff54004eb0 con 0x7eff700b1b00 2024-03-25T19:23:46.770 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.770+0000 7eff667fc640 1 -- 172.21.15.6:0/3682870313 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff701d79d0 con 0x7eff700b1b00 2024-03-25T19:23:46.771 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.770+0000 7eff667fc640 1 -- 172.21.15.6:0/3682870313 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7eff6c001830 con 0x7eff700b1b00 2024-03-25T19:23:46.771 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.770+0000 7eff667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:46.771 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.770+0000 7eff667fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:23:46.771 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.770+0000 7eff667fc640 10 monclient(hunting): dump: 2024-03-25T19:23:46.771 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:23:46.771 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:46.771 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:46.771 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:46.771 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:46.771 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:46.771 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:23:46.771 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:23:46.771 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:23:46.771 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:46.771 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.770+0000 7eff667fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:23:46.776 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.776+0000 7eff667fc640 1 -- 172.21.15.6:0/3682870313 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 724359144 0 0) 0x7eff70136f90 con 0x7eff700b44b0 2024-03-25T19:23:46.776 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.776+0000 7eff667fc640 1 -- 172.21.15.6:0/3682870313 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff54004eb0 con 0x7eff700b44b0 2024-03-25T19:23:46.776 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.776+0000 7eff667fc640 1 -- 172.21.15.6:0/3682870313 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7eff60001ae0 con 0x7eff700b44b0 2024-03-25T19:23:46.776 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.776+0000 7eff667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:46.776 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.776+0000 7eff667fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:23:46.776 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.776+0000 7eff667fc640 10 monclient(hunting): dump: 2024-03-25T19:23:46.776 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:23:46.777 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:46.777 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:46.777 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:46.777 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:46.777 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:46.777 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:23:46.777 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:23:46.777 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:23:46.777 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:46.788 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.788+0000 7eff667fc640 1 -- 172.21.15.6:0/3682870313 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 406625064 0 0) 0x7eff70137e60 con 0x7eff700b4e70 2024-03-25T19:23:46.788 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.788+0000 7eff667fc640 1 -- 172.21.15.6:0/3682870313 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff70136f90 con 0x7eff700b4e70 2024-03-25T19:23:46.788 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.788+0000 7eff667fc640 1 -- 172.21.15.6:0/3682870313 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7eff68001c70 con 0x7eff700b4e70 2024-03-25T19:23:46.788 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.788+0000 7eff667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:46.788 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.788+0000 7eff667fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:23:46.788 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.788+0000 7eff667fc640 10 monclient(hunting): dump: 2024-03-25T19:23:46.788 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:23:46.788 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:46.788 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:46.788 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:46.789 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:46.789 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:46.789 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:23:46.789 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:23:46.789 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:23:46.789 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:46.790 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.789+0000 7eff667fc640 1 -- 172.21.15.6:0/3682870313 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2983876698 0 0) 0x7eff701d79d0 con 0x7eff700b1b00 2024-03-25T19:23:46.790 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.789+0000 7eff667fc640 1 -- 172.21.15.6:0/3682870313 >> v1:172.21.15.6:6790/0 conn(0x7eff700b44b0 legacy=0x7eff701d4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:46.790 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.789+0000 7eff667fc640 1 -- 172.21.15.6:0/3682870313 >> v1:172.21.15.192:6789/0 conn(0x7eff700b4e70 legacy=0x7eff701d7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:46.791 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.789+0000 7eff667fc640 10 monclient: _finish_hunting 0 2024-03-25T19:23:46.791 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.789+0000 7eff667fc640 1 monclient: found mon.a 2024-03-25T19:23:46.791 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.789+0000 7eff667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:46.791 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.789+0000 7eff667fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:23:46.791 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.789+0000 7eff667fc640 1 -- 172.21.15.6:0/3682870313 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff70138140 con 0x7eff700b1b00 2024-03-25T19:23:46.791 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.789+0000 7eff667fc640 10 monclient: _finish_auth 0 2024-03-25T19:23:46.791 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.789+0000 7eff667fc640 10 monclient: _check_auth_tickets 2024-03-25T19:23:46.791 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.789+0000 7eff667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:46.791 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.789+0000 7eff78177640 5 monclient: authenticate success, global_id 4659 2024-03-25T19:23:46.791 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.789+0000 7eff78177640 10 monclient: _renew_subs 2024-03-25T19:23:46.791 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.789+0000 7eff78177640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:23:46.791 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.789+0000 7eff78177640 1 -- 172.21.15.6:0/3682870313 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff701d4540 con 0x7eff700b1b00 2024-03-25T19:23:46.791 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.789+0000 7eff78177640 10 client.4659.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:23:46.791 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.789+0000 7eff78177640 10 monclient: _renew_subs 2024-03-25T19:23:46.792 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.789+0000 7eff78177640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:23:46.792 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.789+0000 7eff78177640 1 -- 172.21.15.6:0/3682870313 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7eff701363f0 con 0x7eff700b1b00 2024-03-25T19:23:46.792 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.789+0000 7eff78177640 1 librados: init done 2024-03-25T19:23:46.792 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.790+0000 7eff78177640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:46.792 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.790+0000 7eff78177640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:46.792 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.790+0000 7eff78177640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:23:46.792 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:46.790+0000 7eff78177640 1 -- 172.21.15.6:0/3682870313 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff7004f730 con 0x7eff700b1b00 2024-03-25T19:23:47.103 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:47.103+0000 7eff667fc640 1 -- 172.21.15.6:0/3682870313 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7eff6c001ce0 con 0x7eff700b1b00 2024-03-25T19:23:47.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:47.103+0000 7eff667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:47.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:47.103+0000 7eff667fc640 1 -- 172.21.15.6:0/3682870313 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7eff6c004980 con 0x7eff700b1b00 2024-03-25T19:23:47.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:47.103+0000 7eff667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:47.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:47.103+0000 7eff667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:23:47.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:47.103+0000 7eff667fc640 10 monclient: dump: 2024-03-25T19:23:47.104 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:23:47.104 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:47.104 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:47.104 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:47.104 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:47.104 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:47.104 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:23:47.105 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:23:47.105 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:23:47.105 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:47.105 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:47.103+0000 7eff67fff640 4 set_mon_vals no callback set 2024-03-25T19:23:47.105 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:47.104+0000 7eff667fc640 1 -- 172.21.15.6:0/3682870313 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7eff6c01aa10 con 0x7eff700b1b00 2024-03-25T19:23:47.105 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:47.105+0000 7eff75712640 10 monclient: get_auth_request con 0x7eff5406e0b0 auth_method 0 2024-03-25T19:23:47.106 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:47.105+0000 7eff667fc640 10 client.4659.objecter ms_handle_connect 0x7eff5406e0b0 2024-03-25T19:23:47.111 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:47.111+0000 7eff667fc640 1 -- 172.21.15.6:0/3682870313 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7eff6c0547c0 con 0x7eff700b1b00 2024-03-25T19:23:47.111 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:47.111+0000 7eff667fc640 10 client.4659.objecter ms_dispatch 0x7eff70000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:23:47.111 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:47.111+0000 7eff667fc640 3 client.4659.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:23:47.111 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:47.111+0000 7eff667fc640 3 client.4659.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:23:47.111 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:47.111+0000 7eff667fc640 20 client.4659.objecter dump_active .. 0 homeless 2024-03-25T19:23:47.111 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:47.111+0000 7eff667fc640 1 -- 172.21.15.6:0/3682870313 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7eff6c050100 con 0x7eff700b1b00 2024-03-25T19:23:47.111 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:47.111+0000 7eff667fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:47.111 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:47.111+0000 7eff667fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:23:47.338 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:47.338+0000 7eff78177640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:47.338 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:47.338+0000 7eff78177640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:47.338 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:47.338+0000 7eff78177640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:23:47.338 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:47.338+0000 7eff78177640 1 -- 172.21.15.6:0/3682870313 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7eff701d7b60 con 0x7eff700b1b00 2024-03-25T19:23:47.367 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:47.367+0000 7eff667fc640 1 -- 172.21.15.6:0/3682870313 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7eff6c052200 con 0x7eff700b1b00 2024-03-25T19:23:47.367 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:47.367+0000 7eff667fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:47.367 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:47.367+0000 7eff667fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:23:47.367 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:23:47.367 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:23:47.370 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:47.369+0000 7eff78177640 10 librados: watch_flush enter 2024-03-25T19:23:47.370 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:47.369+0000 7eff78177640 10 librados: watch_flush exit 2024-03-25T19:23:47.370 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:47.369+0000 7eff78177640 20 client.4659.objecter shutdown clearing up homeless session... 2024-03-25T19:23:47.370 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:47.369+0000 7eff78177640 10 client.4659.objecter successfully canceled tick 2024-03-25T19:23:47.371 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:47.370+0000 7eff78177640 1 -- 172.21.15.6:0/3682870313 >> v1:172.21.15.6:6800/2936652650 conn(0x7eff5406e0b0 legacy=0x7eff54070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:47.371 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:47.370+0000 7eff78177640 10 monclient: shutdown 2024-03-25T19:23:47.371 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:47.370+0000 7eff78177640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:47.371 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:47.370+0000 7eff78177640 1 -- 172.21.15.6:0/3682870313 >> v1:172.21.15.6:6789/0 conn(0x7eff700b1b00 legacy=0x7eff70136880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:47.371 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:47.370+0000 7eff78177640 1 -- 172.21.15.6:0/3682870313 shutdown_connections 2024-03-25T19:23:47.371 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:47.370+0000 7eff78177640 1 -- 172.21.15.6:0/3682870313 >> 172.21.15.6:0/3682870313 conn(0x7eff700617a0 msgr2=0x7eff700b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:47.371 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:47.370+0000 7eff78177640 1 -- 172.21.15.6:0/3682870313 shutdown_connections 2024-03-25T19:23:47.371 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:47.370+0000 7eff78177640 1 -- 172.21.15.6:0/3682870313 wait complete. 2024-03-25T19:23:47.371 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:47.370+0000 7eff78177640 1 librados: shutdown 2024-03-25T19:23:47.381 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:23:49.324 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:10:06.670 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:49.336 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:10:06.683 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:49.362 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:10:06.709 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:49.386 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:10:06.704 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:49.391 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:10:06.738 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:49.392 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:10:06.710 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:49.399 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:10:06.746 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:49.436 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:10:06.754 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:49.437 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:10:06.784 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:54.187 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:23:54.302 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.302+0000 7f8c8ebcf640 10 monclient: get_monmap_and_config 2024-03-25T19:23:54.302 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.302+0000 7f8c8ebcf640 10 monclient: build_initial_monmap 2024-03-25T19:23:54.302 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.302+0000 7f8c8ebcf640 10 monclient: monmap: 2024-03-25T19:23:54.302 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:23:54.302 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:54.302 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:23:54.303181+0000 2024-03-25T19:23:54.302 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:23:54.303181+0000 2024-03-25T19:23:54.302 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:54.302 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:54.303 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:23:54.303 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:23:54.303 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:23:54.303 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:54.304 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.302+0000 7f8c8ebcf640 1 Processor -- start 2024-03-25T19:23:54.304 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.303+0000 7f8c8ebcf640 1 -- start start 2024-03-25T19:23:54.304 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.303+0000 7f8c8ebcf640 10 monclient: init 2024-03-25T19:23:54.304 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.303+0000 7f8c8ebcf640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:54.304 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.303+0000 7f8c8ebcf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:23:54.304 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.303+0000 7f8c8ebcf640 10 monclient(hunting): picked mon.noname-b con 0x7f8c880b4e70 addr v1:172.21.15.6:6790/0 2024-03-25T19:23:54.304 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.303+0000 7f8c8ebcf640 10 monclient(hunting): picked mon.noname-c con 0x7f8c880b44b0 addr v1:172.21.15.192:6789/0 2024-03-25T19:23:54.304 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.303+0000 7f8c8ebcf640 10 monclient(hunting): picked mon.noname-a con 0x7f8c880b1b00 addr v1:172.21.15.6:6789/0 2024-03-25T19:23:54.304 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.303+0000 7f8c8ebcf640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c880a96f0 con 0x7f8c880b1b00 2024-03-25T19:23:54.304 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.304+0000 7f8c8ebcf640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c880ab0e0 con 0x7f8c880b44b0 2024-03-25T19:23:54.304 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.304+0000 7f8c8ebcf640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c88145f70 con 0x7f8c880b4e70 2024-03-25T19:23:54.305 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.304+0000 7f8c8ebcf640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:54.305 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.304+0000 7f8c8ebcf640 10 monclient(hunting): authenticate will time out at 6538.543808s 2024-03-25T19:23:54.312 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.312+0000 7f8c87fff640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f8c880b44b0 0x7f8c88145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:40322/0 (socket says 172.21.15.6:40322) 2024-03-25T19:23:54.312 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.312+0000 7f8c87fff640 1 -- 172.21.15.6:0/3921693217 learned_addr learned my addr 172.21.15.6:0/3921693217 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:23:54.333 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.334+0000 7f8c877fe640 1 -- 172.21.15.6:0/3921693217 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3882566959 0 0) 0x7f8c880ab0e0 con 0x7f8c880b44b0 2024-03-25T19:23:54.334 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.334+0000 7f8c877fe640 10 monclient(hunting): _negotiate 2024-03-25T19:23:54.334 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.334+0000 7f8c877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:54.334 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.334+0000 7f8c877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:54.334 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.334+0000 7f8c877fe640 10 monclient(hunting): my global_id is 4672 2024-03-25T19:23:54.334 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.334+0000 7f8c877fe640 1 -- 172.21.15.6:0/3921693217 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c78003520 con 0x7f8c880b44b0 2024-03-25T19:23:54.339 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.339+0000 7f8c877fe640 1 -- 172.21.15.6:0/3921693217 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 49446360 0 0) 0x7f8c88145f70 con 0x7f8c880b4e70 2024-03-25T19:23:54.339 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.339+0000 7f8c877fe640 10 monclient(hunting): _negotiate 2024-03-25T19:23:54.339 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.339+0000 7f8c877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:54.339 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.339+0000 7f8c877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:54.339 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.339+0000 7f8c877fe640 10 monclient(hunting): my global_id is 4664 2024-03-25T19:23:54.339 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.339+0000 7f8c877fe640 1 -- 172.21.15.6:0/3921693217 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c880ab0e0 con 0x7f8c880b4e70 2024-03-25T19:23:54.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.342+0000 7f8c877fe640 1 -- 172.21.15.6:0/3921693217 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3189408486 0 0) 0x7f8c880a96f0 con 0x7f8c880b1b00 2024-03-25T19:23:54.343 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.343+0000 7f8c877fe640 10 monclient(hunting): _negotiate 2024-03-25T19:23:54.343 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.343+0000 7f8c877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:54.343 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.343+0000 7f8c877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:54.343 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.343+0000 7f8c877fe640 10 monclient(hunting): my global_id is 4662 2024-03-25T19:23:54.343 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.343+0000 7f8c877fe640 1 -- 172.21.15.6:0/3921693217 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c88145f70 con 0x7f8c880b1b00 2024-03-25T19:23:54.349 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.348+0000 7f8c877fe640 1 -- 172.21.15.6:0/3921693217 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2917051648 0 0) 0x7f8c880ab0e0 con 0x7f8c880b4e70 2024-03-25T19:23:54.349 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.348+0000 7f8c877fe640 1 -- 172.21.15.6:0/3921693217 >> v1:172.21.15.192:6789/0 conn(0x7f8c880b44b0 legacy=0x7f8c88145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:54.349 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.348+0000 7f8c877fe640 1 -- 172.21.15.6:0/3921693217 >> v1:172.21.15.6:6789/0 conn(0x7f8c880b1b00 legacy=0x7f8c8814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:54.349 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.348+0000 7f8c877fe640 10 monclient: _finish_hunting 0 2024-03-25T19:23:54.349 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.348+0000 7f8c877fe640 1 monclient: found mon.noname-b 2024-03-25T19:23:54.349 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.348+0000 7f8c877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:54.349 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.348+0000 7f8c877fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-03-25T19:23:54.349 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.348+0000 7f8c877fe640 1 -- 172.21.15.6:0/3921693217 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c880a9b80 con 0x7f8c880b4e70 2024-03-25T19:23:54.349 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.348+0000 7f8c877fe640 10 monclient: _finish_auth 0 2024-03-25T19:23:54.349 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.348+0000 7f8c877fe640 10 monclient: _check_auth_tickets 2024-03-25T19:23:54.349 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.348+0000 7f8c877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:54.349 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.348+0000 7f8c877fe640 1 -- 172.21.15.6:0/3921693217 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f8c7c001670 con 0x7f8c880b4e70 2024-03-25T19:23:54.349 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.348+0000 7f8c877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:54.349 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.348+0000 7f8c877fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:23:54.349 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.348+0000 7f8c877fe640 10 monclient: dump: 2024-03-25T19:23:54.350 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:23:54.350 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:54.350 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:54.350 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:54.350 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:54.350 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:54.350 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:23:54.350 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:23:54.350 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:23:54.350 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:54.350 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.348+0000 7f8c8ebcf640 5 monclient: authenticate success, global_id 4664 2024-03-25T19:23:54.350 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.348+0000 7f8c8ebcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:23:54.372 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.372+0000 7f8c877fe640 1 -- 172.21.15.6:0/3921693217 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c7c0017d0 con 0x7f8c880b4e70 2024-03-25T19:23:54.372 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.372+0000 7f8c877fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:54.372 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.372+0000 7f8c877fe640 1 -- 172.21.15.6:0/3921693217 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f8c7c001be0 con 0x7f8c880b4e70 2024-03-25T19:23:54.372 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.372+0000 7f8c8ebcf640 10 monclient: get_monmap_and_config success 2024-03-25T19:23:54.373 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.372+0000 7f8c8ebcf640 4 set_mon_vals no callback set 2024-03-25T19:23:54.373 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.372+0000 7f8c877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:54.373 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.372+0000 7f8c877fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:23:54.373 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.372+0000 7f8c877fe640 10 monclient: dump: 2024-03-25T19:23:54.373 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:23:54.373 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:54.373 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:54.373 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:54.373 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:54.373 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:54.373 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:23:54.373 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:23:54.373 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:23:54.373 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:54.376 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.373+0000 7f8c8ebcf640 10 monclient: shutdown 2024-03-25T19:23:54.376 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.373+0000 7f8c8ebcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:54.376 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.373+0000 7f8c8ebcf640 1 -- 172.21.15.6:0/3921693217 >> v1:172.21.15.6:6790/0 conn(0x7f8c880b4e70 legacy=0x7f8c8813f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:54.376 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.373+0000 7f8c8ebcf640 1 -- 172.21.15.6:0/3921693217 shutdown_connections 2024-03-25T19:23:54.376 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.373+0000 7f8c8ebcf640 1 -- 172.21.15.6:0/3921693217 >> 172.21.15.6:0/3921693217 conn(0x7f8c880617a0 msgr2=0x7f8c88061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:54.376 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.373+0000 7f8c8ebcf640 1 -- 172.21.15.6:0/3921693217 shutdown_connections 2024-03-25T19:23:54.376 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.373+0000 7f8c8ebcf640 1 -- 172.21.15.6:0/3921693217 wait complete. 2024-03-25T19:23:54.376 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.374+0000 7f8c8ebcf640 10 monclient: build_initial_monmap 2024-03-25T19:23:54.376 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.374+0000 7f8c8ebcf640 10 monclient: monmap: 2024-03-25T19:23:54.376 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:23:54.376 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:23:54.376 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:23:54.377 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:23:54.377 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:54.377 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:23:54.377 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:23:54.377 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:23:54.377 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:23:54.377 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:54.377 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.374+0000 7f8c8ebcf640 1 librados: starting msgr at 2024-03-25T19:23:54.377 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.374+0000 7f8c8ebcf640 1 librados: starting objecter 2024-03-25T19:23:54.377 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.374+0000 7f8c8ebcf640 1 Processor -- start 2024-03-25T19:23:54.377 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.374+0000 7f8c8ebcf640 1 -- start start 2024-03-25T19:23:54.377 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.374+0000 7f8c8ebcf640 1 librados: setting wanted keys 2024-03-25T19:23:54.377 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.374+0000 7f8c8ebcf640 1 librados: calling monclient init 2024-03-25T19:23:54.377 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.374+0000 7f8c8ebcf640 10 monclient: init 2024-03-25T19:23:54.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.375+0000 7f8c8ebcf640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:54.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.375+0000 7f8c8ebcf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:23:54.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.375+0000 7f8c8ebcf640 10 monclient(hunting): picked mon.noname-a con 0x7f8c880b1b00 addr v1:172.21.15.6:6789/0 2024-03-25T19:23:54.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.375+0000 7f8c8ebcf640 10 monclient(hunting): picked mon.noname-b con 0x7f8c880b44b0 addr v1:172.21.15.192:6789/0 2024-03-25T19:23:54.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.375+0000 7f8c8ebcf640 10 monclient(hunting): picked mon.noname-c con 0x7f8c880b4e70 addr v1:172.21.15.6:6790/0 2024-03-25T19:23:54.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.375+0000 7f8c8ebcf640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c88143000 con 0x7f8c880b1b00 2024-03-25T19:23:54.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.375+0000 7f8c8ebcf640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c88139b60 con 0x7f8c880b44b0 2024-03-25T19:23:54.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.375+0000 7f8c8ebcf640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c88137e00 con 0x7f8c880b4e70 2024-03-25T19:23:54.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.375+0000 7f8c8ebcf640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:54.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.375+0000 7f8c8ebcf640 10 monclient(hunting): authenticate will time out at 6538.615390s 2024-03-25T19:23:54.385 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.385+0000 7f8c87fff640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f8c880b44b0 0x7f8c88138850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:40370/0 (socket says 172.21.15.6:40370) 2024-03-25T19:23:54.385 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.385+0000 7f8c87fff640 1 -- 172.21.15.6:0/3777295050 learned_addr learned my addr 172.21.15.6:0/3777295050 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:23:54.389 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.389+0000 7f8c857fa640 10 client.?.objecter ms_handle_connect 0x7f8c880b44b0 2024-03-25T19:23:54.389 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.389+0000 7f8c857fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:54.398 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.398+0000 7f8c857fa640 10 client.?.objecter ms_handle_connect 0x7f8c880b4e70 2024-03-25T19:23:54.398 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.398+0000 7f8c857fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:54.399 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.399+0000 7f8c857fa640 1 -- 172.21.15.6:0/3777295050 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1679707598 0 0) 0x7f8c88139b60 con 0x7f8c880b44b0 2024-03-25T19:23:54.399 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.399+0000 7f8c857fa640 10 monclient(hunting): _negotiate 2024-03-25T19:23:54.399 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.399+0000 7f8c857fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:54.399 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.399+0000 7f8c857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:54.399 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.399+0000 7f8c857fa640 10 monclient(hunting): my global_id is 4675 2024-03-25T19:23:54.399 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.399+0000 7f8c857fa640 1 -- 172.21.15.6:0/3777295050 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c6c004eb0 con 0x7f8c880b44b0 2024-03-25T19:23:54.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.408+0000 7f8c857fa640 1 -- 172.21.15.6:0/3777295050 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 691142629 0 0) 0x7f8c6c004eb0 con 0x7f8c880b44b0 2024-03-25T19:23:54.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.408+0000 7f8c857fa640 1 -- 172.21.15.6:0/3777295050 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c88139b60 con 0x7f8c880b44b0 2024-03-25T19:23:54.410 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.410+0000 7f8c857fa640 1 -- 172.21.15.6:0/3777295050 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f8c740019b0 con 0x7f8c880b44b0 2024-03-25T19:23:54.410 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.410+0000 7f8c857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:54.411 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.410+0000 7f8c857fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:23:54.411 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.410+0000 7f8c857fa640 10 monclient(hunting): dump: 2024-03-25T19:23:54.411 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:23:54.411 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:54.411 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:54.411 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:54.411 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:54.411 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:54.411 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:23:54.411 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:23:54.411 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:23:54.411 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:54.411 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.410+0000 7f8c857fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:23:54.414 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.414+0000 7f8c857fa640 10 client.?.objecter ms_handle_connect 0x7f8c880b1b00 2024-03-25T19:23:54.414 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.414+0000 7f8c857fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:54.421 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.421+0000 7f8c857fa640 1 -- 172.21.15.6:0/3777295050 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 49969447 0 0) 0x7f8c88137e00 con 0x7f8c880b4e70 2024-03-25T19:23:54.421 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.421+0000 7f8c857fa640 10 monclient(hunting): _negotiate 2024-03-25T19:23:54.421 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.421+0000 7f8c857fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:54.422 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.421+0000 7f8c857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:54.422 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.421+0000 7f8c857fa640 10 monclient(hunting): my global_id is 4667 2024-03-25T19:23:54.422 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.421+0000 7f8c857fa640 1 -- 172.21.15.6:0/3777295050 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c6c004eb0 con 0x7f8c880b4e70 2024-03-25T19:23:54.423 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.422+0000 7f8c857fa640 1 -- 172.21.15.6:0/3777295050 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1744313325 0 0) 0x7f8c88139b60 con 0x7f8c880b44b0 2024-03-25T19:23:54.423 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.422+0000 7f8c857fa640 1 -- 172.21.15.6:0/3777295050 >> v1:172.21.15.6:6790/0 conn(0x7f8c880b4e70 legacy=0x7f8c88138f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:54.423 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.422+0000 7f8c857fa640 1 -- 172.21.15.6:0/3777295050 >> v1:172.21.15.6:6789/0 conn(0x7f8c880b1b00 legacy=0x7f8c881418c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:54.423 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.422+0000 7f8c857fa640 10 monclient: _finish_hunting 0 2024-03-25T19:23:54.423 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.422+0000 7f8c857fa640 1 monclient: found mon.b 2024-03-25T19:23:54.423 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.422+0000 7f8c857fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:54.424 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.422+0000 7f8c857fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:23:54.424 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.422+0000 7f8c857fa640 1 -- 172.21.15.6:0/3777295050 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c881380e0 con 0x7f8c880b44b0 2024-03-25T19:23:54.424 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.422+0000 7f8c857fa640 10 monclient: _finish_auth 0 2024-03-25T19:23:54.424 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.422+0000 7f8c857fa640 10 monclient: _check_auth_tickets 2024-03-25T19:23:54.424 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.422+0000 7f8c857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:54.424 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.422+0000 7f8c8ebcf640 5 monclient: authenticate success, global_id 4675 2024-03-25T19:23:54.424 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.422+0000 7f8c8ebcf640 10 monclient: _renew_subs 2024-03-25T19:23:54.424 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.422+0000 7f8c8ebcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:23:54.424 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.422+0000 7f8c8ebcf640 1 -- 172.21.15.6:0/3777295050 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c881383c0 con 0x7f8c880b44b0 2024-03-25T19:23:54.424 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.422+0000 7f8c8ebcf640 10 client.4675.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:23:54.424 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.422+0000 7f8c8ebcf640 10 monclient: _renew_subs 2024-03-25T19:23:54.424 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.422+0000 7f8c8ebcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:23:54.424 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.422+0000 7f8c8ebcf640 1 -- 172.21.15.6:0/3777295050 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c88135d60 con 0x7f8c880b44b0 2024-03-25T19:23:54.424 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.422+0000 7f8c8ebcf640 1 librados: init done 2024-03-25T19:23:54.425 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.423+0000 7f8c8ebcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:54.425 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.423+0000 7f8c8ebcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:54.425 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.423+0000 7f8c8ebcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:23:54.425 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.423+0000 7f8c8ebcf640 1 -- 172.21.15.6:0/3777295050 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c8804f730 con 0x7f8c880b44b0 2024-03-25T19:23:54.711 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.711+0000 7f8c857fa640 1 -- 172.21.15.6:0/3777295050 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c74004860 con 0x7f8c880b44b0 2024-03-25T19:23:54.711 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.711+0000 7f8c857fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:54.711 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.711+0000 7f8c857fa640 1 -- 172.21.15.6:0/3777295050 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f8c74004c70 con 0x7f8c880b44b0 2024-03-25T19:23:54.711 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.711+0000 7f8c857fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:54.711 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.711+0000 7f8c857fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:23:54.711 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.711+0000 7f8c857fa640 10 monclient: dump: 2024-03-25T19:23:54.711 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:23:54.712 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:23:54.712 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:54.712 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:23:54.712 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:54.712 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:23:54.712 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:23:54.712 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:23:54.712 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:23:54.712 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:23:54.712 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.711+0000 7f8c86ffd640 4 set_mon_vals no callback set 2024-03-25T19:23:54.713 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.712+0000 7f8c857fa640 1 -- 172.21.15.6:0/3777295050 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f8c74004e90 con 0x7f8c880b44b0 2024-03-25T19:23:54.713 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.712+0000 7f8c8c96b640 10 monclient: get_auth_request con 0x7f8c6c06dd40 auth_method 0 2024-03-25T19:23:54.713 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.713+0000 7f8c857fa640 10 client.4675.objecter ms_handle_connect 0x7f8c6c06dd40 2024-03-25T19:23:54.715 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.715+0000 7f8c857fa640 1 -- 172.21.15.6:0/3777295050 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f8c74054920 con 0x7f8c880b44b0 2024-03-25T19:23:54.715 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.715+0000 7f8c857fa640 10 client.4675.objecter ms_dispatch 0x7f8c88000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:23:54.715 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.715+0000 7f8c857fa640 3 client.4675.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:23:54.716 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.715+0000 7f8c857fa640 3 client.4675.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:23:54.716 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.715+0000 7f8c857fa640 20 client.4675.objecter dump_active .. 0 homeless 2024-03-25T19:23:54.718 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.718+0000 7f8c857fa640 1 -- 172.21.15.6:0/3777295050 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8c74050430 con 0x7f8c880b44b0 2024-03-25T19:23:54.718 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.718+0000 7f8c857fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:54.718 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.718+0000 7f8c857fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:23:54.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.958+0000 7f8c8ebcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:54.959 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.958+0000 7f8c8ebcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:54.959 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.958+0000 7f8c8ebcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:23:54.959 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.958+0000 7f8c8ebcf640 1 -- 172.21.15.6:0/3777295050 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c880aa090 con 0x7f8c880b44b0 2024-03-25T19:23:54.986 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.986+0000 7f8c857fa640 1 -- 172.21.15.6:0/3777295050 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f8c74052360 con 0x7f8c880b44b0 2024-03-25T19:23:54.986 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.986+0000 7f8c857fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:54.986 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.986+0000 7f8c857fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:23:54.986 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:23:54.987 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:23:54.989 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.988+0000 7f8c8ebcf640 10 librados: watch_flush enter 2024-03-25T19:23:54.989 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.989+0000 7f8c8ebcf640 10 librados: watch_flush exit 2024-03-25T19:23:54.989 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.989+0000 7f8c8ebcf640 20 client.4675.objecter shutdown clearing up homeless session... 2024-03-25T19:23:54.990 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.989+0000 7f8c8ebcf640 10 client.4675.objecter successfully canceled tick 2024-03-25T19:23:54.990 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.989+0000 7f8c8ebcf640 1 -- 172.21.15.6:0/3777295050 >> v1:172.21.15.6:6800/2936652650 conn(0x7f8c6c06dd40 legacy=0x7f8c6c070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:54.990 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.989+0000 7f8c8ebcf640 10 monclient: shutdown 2024-03-25T19:23:54.990 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.989+0000 7f8c8ebcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:54.990 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.989+0000 7f8c8ebcf640 1 -- 172.21.15.6:0/3777295050 >> v1:172.21.15.192:6789/0 conn(0x7f8c880b44b0 legacy=0x7f8c88138850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:54.990 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.989+0000 7f8c8ebcf640 1 -- 172.21.15.6:0/3777295050 shutdown_connections 2024-03-25T19:23:54.990 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.989+0000 7f8c8ebcf640 1 -- 172.21.15.6:0/3777295050 >> 172.21.15.6:0/3777295050 conn(0x7f8c880617a0 msgr2=0x7f8c8813eb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:54.990 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.989+0000 7f8c8ebcf640 1 -- 172.21.15.6:0/3777295050 shutdown_connections 2024-03-25T19:23:54.990 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.989+0000 7f8c8ebcf640 1 -- 172.21.15.6:0/3777295050 wait complete. 2024-03-25T19:23:54.990 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:23:54.989+0000 7f8c8ebcf640 1 librados: shutdown 2024-03-25T19:23:55.002 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:24:01.808 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:24:01.930 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.930+0000 7f00e0b4d640 10 monclient: get_monmap_and_config 2024-03-25T19:24:01.930 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.930+0000 7f00e0b4d640 10 monclient: build_initial_monmap 2024-03-25T19:24:01.930 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.930+0000 7f00e0b4d640 10 monclient: monmap: 2024-03-25T19:24:01.930 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:24:01.930 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:01.930 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:24:01.931346+0000 2024-03-25T19:24:01.930 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:24:01.931346+0000 2024-03-25T19:24:01.930 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:01.931 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:01.931 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:24:01.931 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:24:01.931 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:24:01.931 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:01.932 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.930+0000 7f00e0b4d640 1 Processor -- start 2024-03-25T19:24:01.932 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.931+0000 7f00e0b4d640 1 -- start start 2024-03-25T19:24:01.932 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.931+0000 7f00e0b4d640 10 monclient: init 2024-03-25T19:24:01.932 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.931+0000 7f00e0b4d640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:01.932 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.931+0000 7f00e0b4d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:24:01.932 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.931+0000 7f00e0b4d640 10 monclient(hunting): picked mon.noname-c con 0x7f00dc0b4e70 addr v1:172.21.15.192:6789/0 2024-03-25T19:24:01.933 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.931+0000 7f00e0b4d640 10 monclient(hunting): picked mon.noname-a con 0x7f00dc0b44b0 addr v1:172.21.15.6:6789/0 2024-03-25T19:24:01.933 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.931+0000 7f00e0b4d640 10 monclient(hunting): picked mon.noname-b con 0x7f00dc0b1b00 addr v1:172.21.15.6:6790/0 2024-03-25T19:24:01.933 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.931+0000 7f00e0b4d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00dc0a96f0 con 0x7f00dc0b44b0 2024-03-25T19:24:01.933 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.931+0000 7f00e0b4d640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00dc0ab0e0 con 0x7f00dc0b4e70 2024-03-25T19:24:01.933 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.931+0000 7f00e0b4d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00dc145f70 con 0x7f00dc0b1b00 2024-03-25T19:24:01.933 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.931+0000 7f00e0b4d640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:01.933 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.931+0000 7f00e0b4d640 10 monclient(hunting): authenticate will time out at 6546.171697s 2024-03-25T19:24:01.940 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.940+0000 7f00dad9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f00dc0b1b00 0x7f00dc14dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:47148/0 (socket says 172.21.15.6:47148) 2024-03-25T19:24:01.940 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.940+0000 7f00dad9d640 1 -- 172.21.15.6:0/3120389496 learned_addr learned my addr 172.21.15.6:0/3120389496 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:24:01.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.957+0000 7f00d959a640 1 -- 172.21.15.6:0/3120389496 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1322849784 0 0) 0x7f00dc0ab0e0 con 0x7f00dc0b4e70 2024-03-25T19:24:01.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.957+0000 7f00d959a640 10 monclient(hunting): _negotiate 2024-03-25T19:24:01.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.957+0000 7f00d959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:01.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.957+0000 7f00d959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:01.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.958+0000 7f00d959a640 10 monclient(hunting): my global_id is 4678 2024-03-25T19:24:01.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.958+0000 7f00d959a640 1 -- 172.21.15.6:0/3120389496 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00c4003520 con 0x7f00dc0b4e70 2024-03-25T19:24:01.966 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.966+0000 7f00d959a640 1 -- 172.21.15.6:0/3120389496 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2627184930 0 0) 0x7f00dc0a96f0 con 0x7f00dc0b44b0 2024-03-25T19:24:01.966 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.966+0000 7f00d959a640 10 monclient(hunting): _negotiate 2024-03-25T19:24:01.967 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.966+0000 7f00d959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:01.967 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.966+0000 7f00d959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:01.967 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.966+0000 7f00d959a640 10 monclient(hunting): my global_id is 4668 2024-03-25T19:24:01.967 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.966+0000 7f00d959a640 1 -- 172.21.15.6:0/3120389496 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00dc0ab0e0 con 0x7f00dc0b44b0 2024-03-25T19:24:01.968 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.968+0000 7f00d959a640 1 -- 172.21.15.6:0/3120389496 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3532725506 0 0) 0x7f00dc145f70 con 0x7f00dc0b1b00 2024-03-25T19:24:01.968 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.968+0000 7f00d959a640 10 monclient(hunting): _negotiate 2024-03-25T19:24:01.968 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.968+0000 7f00d959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:01.968 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.968+0000 7f00d959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:01.968 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.968+0000 7f00d959a640 10 monclient(hunting): my global_id is 4670 2024-03-25T19:24:01.968 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.968+0000 7f00d959a640 1 -- 172.21.15.6:0/3120389496 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00dc0a96f0 con 0x7f00dc0b1b00 2024-03-25T19:24:01.974 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.974+0000 7f00d959a640 1 -- 172.21.15.6:0/3120389496 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2519805091 0 0) 0x7f00c4003520 con 0x7f00dc0b4e70 2024-03-25T19:24:01.974 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.974+0000 7f00d959a640 1 -- 172.21.15.6:0/3120389496 >> v1:172.21.15.6:6790/0 conn(0x7f00dc0b1b00 legacy=0x7f00dc14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:01.974 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.974+0000 7f00d959a640 1 -- 172.21.15.6:0/3120389496 >> v1:172.21.15.6:6789/0 conn(0x7f00dc0b44b0 legacy=0x7f00dc145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:01.974 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.974+0000 7f00d959a640 10 monclient: _finish_hunting 0 2024-03-25T19:24:01.974 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.974+0000 7f00d959a640 1 monclient: found mon.noname-c 2024-03-25T19:24:01.975 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.974+0000 7f00d959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:01.975 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.974+0000 7f00d959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:24:01.975 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.974+0000 7f00d959a640 1 -- 172.21.15.6:0/3120389496 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00dc0a9b80 con 0x7f00dc0b4e70 2024-03-25T19:24:01.975 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.974+0000 7f00d959a640 10 monclient: _finish_auth 0 2024-03-25T19:24:01.975 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.974+0000 7f00d959a640 10 monclient: _check_auth_tickets 2024-03-25T19:24:01.975 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.974+0000 7f00d959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:01.975 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.974+0000 7f00d959a640 1 -- 172.21.15.6:0/3120389496 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f00cc001690 con 0x7f00dc0b4e70 2024-03-25T19:24:01.975 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.974+0000 7f00d959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:01.975 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.974+0000 7f00d959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:24:01.975 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.974+0000 7f00d959a640 10 monclient: dump: 2024-03-25T19:24:01.975 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:01.975 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:01.975 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:01.975 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:01.975 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:01.976 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:01.976 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:01.976 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:01.976 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:01.976 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:01.976 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.974+0000 7f00e0b4d640 5 monclient: authenticate success, global_id 4678 2024-03-25T19:24:01.976 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.974+0000 7f00e0b4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:01.982 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.982+0000 7f00d959a640 1 -- 172.21.15.6:0/3120389496 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00cc0017f0 con 0x7f00dc0b4e70 2024-03-25T19:24:01.982 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.982+0000 7f00d959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:01.982 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.982+0000 7f00d959a640 1 -- 172.21.15.6:0/3120389496 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f00cc001c00 con 0x7f00dc0b4e70 2024-03-25T19:24:01.983 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.982+0000 7f00d959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:01.983 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.982+0000 7f00d959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:24:01.983 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.982+0000 7f00d959a640 10 monclient: dump: 2024-03-25T19:24:01.983 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:01.983 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:01.983 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:01.983 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:01.983 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:01.983 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:01.983 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:01.983 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:01.983 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:01.983 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:01.983 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.982+0000 7f00e0b4d640 10 monclient: get_monmap_and_config success 2024-03-25T19:24:01.983 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.982+0000 7f00e0b4d640 4 set_mon_vals no callback set 2024-03-25T19:24:01.985 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.983+0000 7f00e0b4d640 10 monclient: shutdown 2024-03-25T19:24:01.985 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.983+0000 7f00e0b4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:01.985 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.983+0000 7f00e0b4d640 1 -- 172.21.15.6:0/3120389496 >> v1:172.21.15.192:6789/0 conn(0x7f00dc0b4e70 legacy=0x7f00dc13f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:01.985 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.983+0000 7f00e0b4d640 1 -- 172.21.15.6:0/3120389496 shutdown_connections 2024-03-25T19:24:01.985 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.983+0000 7f00e0b4d640 1 -- 172.21.15.6:0/3120389496 >> 172.21.15.6:0/3120389496 conn(0x7f00dc0617a0 msgr2=0x7f00dc061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:01.985 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.983+0000 7f00e0b4d640 1 -- 172.21.15.6:0/3120389496 shutdown_connections 2024-03-25T19:24:01.985 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.983+0000 7f00e0b4d640 1 -- 172.21.15.6:0/3120389496 wait complete. 2024-03-25T19:24:01.985 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.984+0000 7f00e0b4d640 10 monclient: build_initial_monmap 2024-03-25T19:24:01.985 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.984+0000 7f00e0b4d640 10 monclient: monmap: 2024-03-25T19:24:01.985 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:24:01.985 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:24:01.985 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:24:01.986 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:24:01.986 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:01.986 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:24:01.986 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:24:01.986 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:24:01.986 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:24:01.986 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:01.986 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.985+0000 7f00e0b4d640 1 librados: starting msgr at 2024-03-25T19:24:01.986 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.985+0000 7f00e0b4d640 1 librados: starting objecter 2024-03-25T19:24:01.986 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.985+0000 7f00e0b4d640 1 Processor -- start 2024-03-25T19:24:01.986 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.985+0000 7f00e0b4d640 1 -- start start 2024-03-25T19:24:01.986 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.985+0000 7f00e0b4d640 1 librados: setting wanted keys 2024-03-25T19:24:01.986 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.985+0000 7f00e0b4d640 1 librados: calling monclient init 2024-03-25T19:24:01.987 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.985+0000 7f00e0b4d640 10 monclient: init 2024-03-25T19:24:01.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.986+0000 7f00e0b4d640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:01.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.986+0000 7f00e0b4d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:24:01.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.986+0000 7f00e0b4d640 10 monclient(hunting): picked mon.noname-a con 0x7f00dc0b1b00 addr v1:172.21.15.6:6789/0 2024-03-25T19:24:01.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.986+0000 7f00e0b4d640 10 monclient(hunting): picked mon.noname-b con 0x7f00dc0b44b0 addr v1:172.21.15.192:6789/0 2024-03-25T19:24:01.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.986+0000 7f00e0b4d640 10 monclient(hunting): picked mon.noname-c con 0x7f00dc0b4e70 addr v1:172.21.15.6:6790/0 2024-03-25T19:24:01.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.986+0000 7f00e0b4d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00dc13ce50 con 0x7f00dc0b1b00 2024-03-25T19:24:01.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.986+0000 7f00e0b4d640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00dc13b980 con 0x7f00dc0b44b0 2024-03-25T19:24:01.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.986+0000 7f00e0b4d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00dc141f50 con 0x7f00dc0b4e70 2024-03-25T19:24:01.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.986+0000 7f00e0b4d640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:01.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.986+0000 7f00e0b4d640 10 monclient(hunting): authenticate will time out at 6546.226079s 2024-03-25T19:24:01.996 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.996+0000 7f00da59c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f00dc0b1b00 0x7f00dc1df1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:49204/0 (socket says 172.21.15.6:49204) 2024-03-25T19:24:01.996 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:01.996+0000 7f00da59c640 1 -- 172.21.15.6:0/402478824 learned_addr learned my addr 172.21.15.6:0/402478824 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:24:02.000 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.000+0000 7f00caffd640 10 client.?.objecter ms_handle_connect 0x7f00dc0b44b0 2024-03-25T19:24:02.000 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.000+0000 7f00caffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:02.000 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.000+0000 7f00caffd640 10 client.?.objecter ms_handle_connect 0x7f00dc0b1b00 2024-03-25T19:24:02.000 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.000+0000 7f00caffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:02.014 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.014+0000 7f00caffd640 1 -- 172.21.15.6:0/402478824 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1026979128 0 0) 0x7f00dc13b980 con 0x7f00dc0b44b0 2024-03-25T19:24:02.014 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.014+0000 7f00caffd640 10 monclient(hunting): _negotiate 2024-03-25T19:24:02.014 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.014+0000 7f00caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:02.015 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.014+0000 7f00caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:02.015 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.014+0000 7f00caffd640 10 monclient(hunting): my global_id is 4681 2024-03-25T19:24:02.015 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.014+0000 7f00caffd640 1 -- 172.21.15.6:0/402478824 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00bc004eb0 con 0x7f00dc0b44b0 2024-03-25T19:24:02.015 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.016+0000 7f00caffd640 10 client.?.objecter ms_handle_connect 0x7f00dc0b4e70 2024-03-25T19:24:02.015 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.016+0000 7f00caffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:02.018 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.018+0000 7f00caffd640 1 -- 172.21.15.6:0/402478824 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1094507659 0 0) 0x7f00dc13ce50 con 0x7f00dc0b1b00 2024-03-25T19:24:02.018 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.018+0000 7f00caffd640 10 monclient(hunting): _negotiate 2024-03-25T19:24:02.018 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.018+0000 7f00caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:02.018 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.018+0000 7f00caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:02.018 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.018+0000 7f00caffd640 10 monclient(hunting): my global_id is 4671 2024-03-25T19:24:02.018 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.018+0000 7f00caffd640 1 -- 172.21.15.6:0/402478824 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00dc13b980 con 0x7f00dc0b1b00 2024-03-25T19:24:02.024 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.024+0000 7f00caffd640 1 -- 172.21.15.6:0/402478824 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2059190819 0 0) 0x7f00dc141f50 con 0x7f00dc0b4e70 2024-03-25T19:24:02.024 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.024+0000 7f00caffd640 10 monclient(hunting): _negotiate 2024-03-25T19:24:02.024 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.024+0000 7f00caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:02.024 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.024+0000 7f00caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:02.024 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.024+0000 7f00caffd640 10 monclient(hunting): my global_id is 4673 2024-03-25T19:24:02.024 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.024+0000 7f00caffd640 1 -- 172.21.15.6:0/402478824 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00dc13ce50 con 0x7f00dc0b4e70 2024-03-25T19:24:02.028 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.028+0000 7f00caffd640 1 -- 172.21.15.6:0/402478824 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 825394563 0 0) 0x7f00bc004eb0 con 0x7f00dc0b44b0 2024-03-25T19:24:02.028 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.028+0000 7f00caffd640 1 -- 172.21.15.6:0/402478824 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00dc141f50 con 0x7f00dc0b44b0 2024-03-25T19:24:02.028 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.028+0000 7f00caffd640 1 -- 172.21.15.6:0/402478824 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f00d0001970 con 0x7f00dc0b44b0 2024-03-25T19:24:02.028 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.028+0000 7f00caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:02.028 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.028+0000 7f00caffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:24:02.028 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.028+0000 7f00caffd640 10 monclient(hunting): dump: 2024-03-25T19:24:02.028 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:02.028 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:02.028 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:02.028 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:02.028 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:02.028 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:02.028 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:02.028 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:02.029 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:02.029 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:02.029 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.028+0000 7f00caffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:24:02.034 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.034+0000 7f00caffd640 1 -- 172.21.15.6:0/402478824 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4048632906 0 0) 0x7f00dc13b980 con 0x7f00dc0b1b00 2024-03-25T19:24:02.034 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.034+0000 7f00caffd640 1 -- 172.21.15.6:0/402478824 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00bc004eb0 con 0x7f00dc0b1b00 2024-03-25T19:24:02.034 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.034+0000 7f00caffd640 1 -- 172.21.15.6:0/402478824 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f00cc001b40 con 0x7f00dc0b1b00 2024-03-25T19:24:02.034 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.034+0000 7f00caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:02.034 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.034+0000 7f00caffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:24:02.035 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.034+0000 7f00caffd640 10 monclient(hunting): dump: 2024-03-25T19:24:02.035 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:02.035 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:02.035 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:02.035 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:02.035 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:02.035 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:02.035 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:02.035 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:02.035 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:02.035 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:02.044 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.044+0000 7f00caffd640 1 -- 172.21.15.6:0/402478824 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3346747040 0 0) 0x7f00dc141f50 con 0x7f00dc0b44b0 2024-03-25T19:24:02.044 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.044+0000 7f00caffd640 1 -- 172.21.15.6:0/402478824 >> v1:172.21.15.6:6790/0 conn(0x7f00dc0b4e70 legacy=0x7f00dc1e64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:02.044 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.044+0000 7f00caffd640 1 -- 172.21.15.6:0/402478824 >> v1:172.21.15.6:6789/0 conn(0x7f00dc0b1b00 legacy=0x7f00dc1df1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:02.044 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.044+0000 7f00caffd640 10 monclient: _finish_hunting 0 2024-03-25T19:24:02.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.044+0000 7f00caffd640 1 monclient: found mon.b 2024-03-25T19:24:02.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.044+0000 7f00caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:02.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.044+0000 7f00caffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:24:02.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.044+0000 7f00caffd640 1 -- 172.21.15.6:0/402478824 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00dc1420e0 con 0x7f00dc0b44b0 2024-03-25T19:24:02.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.044+0000 7f00caffd640 10 monclient: _finish_auth 0 2024-03-25T19:24:02.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.044+0000 7f00caffd640 10 monclient: _check_auth_tickets 2024-03-25T19:24:02.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.044+0000 7f00caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:02.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.044+0000 7f00e0b4d640 5 monclient: authenticate success, global_id 4681 2024-03-25T19:24:02.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.044+0000 7f00e0b4d640 10 monclient: _renew_subs 2024-03-25T19:24:02.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.044+0000 7f00e0b4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:24:02.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.044+0000 7f00e0b4d640 1 -- 172.21.15.6:0/402478824 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00dc140160 con 0x7f00dc0b44b0 2024-03-25T19:24:02.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.044+0000 7f00e0b4d640 10 client.4681.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:24:02.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.044+0000 7f00e0b4d640 10 monclient: _renew_subs 2024-03-25T19:24:02.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.044+0000 7f00e0b4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:24:02.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.044+0000 7f00e0b4d640 1 -- 172.21.15.6:0/402478824 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f00dc140970 con 0x7f00dc0b44b0 2024-03-25T19:24:02.046 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.044+0000 7f00e0b4d640 1 librados: init done 2024-03-25T19:24:02.046 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.045+0000 7f00e0b4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:02.046 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.045+0000 7f00e0b4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:02.046 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.045+0000 7f00e0b4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:24:02.046 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.045+0000 7f00e0b4d640 1 -- 172.21.15.6:0/402478824 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00dc04f730 con 0x7f00dc0b44b0 2024-03-25T19:24:02.319 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.319+0000 7f00caffd640 1 -- 172.21.15.6:0/402478824 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00d00035f0 con 0x7f00dc0b44b0 2024-03-25T19:24:02.320 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.319+0000 7f00caffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:02.320 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.319+0000 7f00caffd640 1 -- 172.21.15.6:0/402478824 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f00d0003900 con 0x7f00dc0b44b0 2024-03-25T19:24:02.320 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.319+0000 7f00caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:02.320 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.319+0000 7f00caffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:24:02.320 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.319+0000 7f00caffd640 10 monclient: dump: 2024-03-25T19:24:02.320 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:02.320 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:02.320 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:02.320 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:02.320 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:02.320 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:02.320 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:02.321 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:02.321 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:02.321 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:02.321 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.319+0000 7f00d8d99640 4 set_mon_vals no callback set 2024-03-25T19:24:02.321 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.321+0000 7f00caffd640 1 -- 172.21.15.6:0/402478824 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f00d0019910 con 0x7f00dc0b44b0 2024-03-25T19:24:02.322 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.321+0000 7f00da59c640 10 monclient: get_auth_request con 0x7f00bc06df20 auth_method 0 2024-03-25T19:24:02.322 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.322+0000 7f00caffd640 10 client.4681.objecter ms_handle_connect 0x7f00bc06df20 2024-03-25T19:24:02.324 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.324+0000 7f00caffd640 1 -- 172.21.15.6:0/402478824 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1116577513 0 0) 0x7f00d0053610 con 0x7f00dc0b44b0 2024-03-25T19:24:02.324 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.324+0000 7f00caffd640 10 client.4681.objecter ms_dispatch 0x7f00dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:24:02.325 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.324+0000 7f00caffd640 3 client.4681.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:24:02.325 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.324+0000 7f00caffd640 3 client.4681.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:24:02.325 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.324+0000 7f00caffd640 20 client.4681.objecter dump_active .. 0 homeless 2024-03-25T19:24:02.327 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.327+0000 7f00caffd640 1 -- 172.21.15.6:0/402478824 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f00d004f120 con 0x7f00dc0b44b0 2024-03-25T19:24:02.327 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.327+0000 7f00caffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:02.327 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.327+0000 7f00caffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:24:02.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.559+0000 7f00e0b4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:02.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.559+0000 7f00e0b4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:02.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.559+0000 7f00e0b4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:24:02.559 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.559+0000 7f00e0b4d640 1 -- 172.21.15.6:0/402478824 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00dc1404b0 con 0x7f00dc0b44b0 2024-03-25T19:24:02.615 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.615+0000 7f00caffd640 1 -- 172.21.15.6:0/402478824 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1845389207) 0x7f00d0051050 con 0x7f00dc0b44b0 2024-03-25T19:24:02.615 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.615+0000 7f00caffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:02.615 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.615+0000 7f00caffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:24:02.615 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:24:02.615 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:14:10.388575+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:14:10.388575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:24:02.618 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.617+0000 7f00e0b4d640 10 librados: watch_flush enter 2024-03-25T19:24:02.618 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.618+0000 7f00e0b4d640 10 librados: watch_flush exit 2024-03-25T19:24:02.618 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.618+0000 7f00e0b4d640 20 client.4681.objecter shutdown clearing up homeless session... 2024-03-25T19:24:02.618 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.618+0000 7f00e0b4d640 10 client.4681.objecter successfully canceled tick 2024-03-25T19:24:02.618 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.618+0000 7f00e0b4d640 1 -- 172.21.15.6:0/402478824 >> v1:172.21.15.6:6800/2936652650 conn(0x7f00bc06df20 legacy=0x7f00bc0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:02.618 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.618+0000 7f00e0b4d640 10 monclient: shutdown 2024-03-25T19:24:02.619 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.618+0000 7f00e0b4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:02.619 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.618+0000 7f00e0b4d640 1 -- 172.21.15.6:0/402478824 >> v1:172.21.15.192:6789/0 conn(0x7f00dc0b44b0 legacy=0x7f00dc1df900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:02.619 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.618+0000 7f00e0b4d640 1 -- 172.21.15.6:0/402478824 shutdown_connections 2024-03-25T19:24:02.619 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.618+0000 7f00e0b4d640 1 -- 172.21.15.6:0/402478824 >> 172.21.15.6:0/402478824 conn(0x7f00dc0617a0 msgr2=0x7f00dc0b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:02.619 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.618+0000 7f00e0b4d640 1 -- 172.21.15.6:0/402478824 shutdown_connections 2024-03-25T19:24:02.619 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.618+0000 7f00e0b4d640 1 -- 172.21.15.6:0/402478824 wait complete. 2024-03-25T19:24:02.619 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:02.618+0000 7f00e0b4d640 1 librados: shutdown 2024-03-25T19:24:02.630 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:24:09.437 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:24:09.532 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:10:26.879 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:09.538 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:10:26.885 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:09.542 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:10:26.860 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:09.544 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.544+0000 7f11de455640 10 monclient: get_monmap_and_config 2024-03-25T19:24:09.544 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.544+0000 7f11de455640 10 monclient: build_initial_monmap 2024-03-25T19:24:09.544 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.544+0000 7f11de455640 10 monclient: monmap: 2024-03-25T19:24:09.544 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:24:09.544 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:09.544 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:24:09.545283+0000 2024-03-25T19:24:09.544 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:24:09.545283+0000 2024-03-25T19:24:09.544 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:09.544 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:09.544 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:24:09.544 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:24:09.544 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:24:09.544 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:09.546 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.545+0000 7f11de455640 1 Processor -- start 2024-03-25T19:24:09.546 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.545+0000 7f11de455640 1 -- start start 2024-03-25T19:24:09.546 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.545+0000 7f11de455640 10 monclient: init 2024-03-25T19:24:09.546 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.545+0000 7f11de455640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:09.546 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.545+0000 7f11de455640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:24:09.546 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.545+0000 7f11de455640 10 monclient(hunting): picked mon.noname-b con 0x7f11d806d180 addr v1:172.21.15.6:6790/0 2024-03-25T19:24:09.546 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.545+0000 7f11de455640 10 monclient(hunting): picked mon.noname-a con 0x7f11d806d560 addr v1:172.21.15.6:6789/0 2024-03-25T19:24:09.546 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.545+0000 7f11de455640 10 monclient(hunting): picked mon.noname-c con 0x7f11d80b06f0 addr v1:172.21.15.192:6789/0 2024-03-25T19:24:09.546 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.545+0000 7f11de455640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d8146930 con 0x7f11d806d560 2024-03-25T19:24:09.546 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.545+0000 7f11de455640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d8070710 con 0x7f11d80b06f0 2024-03-25T19:24:09.546 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.545+0000 7f11de455640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d8070980 con 0x7f11d806d180 2024-03-25T19:24:09.546 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.545+0000 7f11de455640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:09.547 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.545+0000 7f11de455640 10 monclient(hunting): authenticate will time out at 6553.785721s 2024-03-25T19:24:09.547 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:10:26.892 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:09.547 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.547+0000 7f11d7fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f11d806d180 0x7f11d806c630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:60816/0 (socket says 172.21.15.6:60816) 2024-03-25T19:24:09.547 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.547+0000 7f11d7fff640 1 -- 172.21.15.6:0/818264058 learned_addr learned my addr 172.21.15.6:0/818264058 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:24:09.548 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:10:26.895 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:09.586 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:10:26.904 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:09.610 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:10:26.957 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:09.619 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.620+0000 7f11d6ffd640 1 -- 172.21.15.6:0/818264058 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2516660340 0 0) 0x7f11d8146930 con 0x7f11d806d560 2024-03-25T19:24:09.619 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.620+0000 7f11d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:24:09.620 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.620+0000 7f11d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:09.620 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.620+0000 7f11d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:09.620 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.620+0000 7f11d6ffd640 10 monclient(hunting): my global_id is 4674 2024-03-25T19:24:09.620 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.620+0000 7f11d6ffd640 1 -- 172.21.15.6:0/818264058 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11c0003520 con 0x7f11d806d560 2024-03-25T19:24:09.623 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:10:26.970 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:09.635 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:10:26.953 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:09.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.723+0000 7f11d6ffd640 1 -- 172.21.15.6:0/818264058 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1899657216 0 0) 0x7f11d8070980 con 0x7f11d806d180 2024-03-25T19:24:09.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.723+0000 7f11d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:24:09.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.723+0000 7f11d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:09.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.723+0000 7f11d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:09.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.723+0000 7f11d6ffd640 10 monclient(hunting): my global_id is 4676 2024-03-25T19:24:09.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.723+0000 7f11d6ffd640 1 -- 172.21.15.6:0/818264058 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11d8146930 con 0x7f11d806d180 2024-03-25T19:24:09.730 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.730+0000 7f11d6ffd640 1 -- 172.21.15.6:0/818264058 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 511249335 0 0) 0x7f11c0003520 con 0x7f11d806d560 2024-03-25T19:24:09.730 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.730+0000 7f11d6ffd640 1 -- 172.21.15.6:0/818264058 >> v1:172.21.15.6:6790/0 conn(0x7f11d806d180 legacy=0x7f11d806c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:09.730 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.730+0000 7f11d6ffd640 1 -- 172.21.15.6:0/818264058 >> v1:172.21.15.192:6789/0 conn(0x7f11d80b06f0 legacy=0x7f11d8146dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:09.730 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.730+0000 7f11d6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:24:09.730 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.730+0000 7f11d6ffd640 1 monclient: found mon.noname-a 2024-03-25T19:24:09.730 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.730+0000 7f11d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:09.730 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.730+0000 7f11d6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-03-25T19:24:09.730 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.730+0000 7f11d6ffd640 1 -- 172.21.15.6:0/818264058 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d81474d0 con 0x7f11d806d560 2024-03-25T19:24:09.731 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.730+0000 7f11d6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:24:09.731 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.730+0000 7f11d6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:24:09.731 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.730+0000 7f11d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:09.731 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.730+0000 7f11d6ffd640 1 -- 172.21.15.6:0/818264058 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f11cc001690 con 0x7f11d806d560 2024-03-25T19:24:09.731 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.730+0000 7f11d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:09.731 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.730+0000 7f11d6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:24:09.731 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.730+0000 7f11d6ffd640 10 monclient: dump: 2024-03-25T19:24:09.731 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:09.731 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:09.731 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:09.731 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:09.731 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:09.731 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:09.731 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:09.731 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:09.731 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:09.732 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:09.732 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.730+0000 7f11de455640 5 monclient: authenticate success, global_id 4674 2024-03-25T19:24:09.732 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.730+0000 7f11de455640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:09.779 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.779+0000 7f11d6ffd640 1 -- 172.21.15.6:0/818264058 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11cc0017f0 con 0x7f11d806d560 2024-03-25T19:24:09.779 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.779+0000 7f11d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:09.779 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.779+0000 7f11d6ffd640 1 -- 172.21.15.6:0/818264058 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f11cc001c00 con 0x7f11d806d560 2024-03-25T19:24:09.780 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.779+0000 7f11d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:09.780 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.779+0000 7f11d6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:24:09.780 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.779+0000 7f11d6ffd640 10 monclient: dump: 2024-03-25T19:24:09.780 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:09.780 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:09.780 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:09.780 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:09.780 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:09.780 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:09.780 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:09.780 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:09.780 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:09.780 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:09.780 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.779+0000 7f11de455640 10 monclient: get_monmap_and_config success 2024-03-25T19:24:09.781 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.779+0000 7f11de455640 4 set_mon_vals no callback set 2024-03-25T19:24:09.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.780+0000 7f11de455640 10 monclient: shutdown 2024-03-25T19:24:09.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.780+0000 7f11de455640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:09.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.780+0000 7f11de455640 1 -- 172.21.15.6:0/818264058 >> v1:172.21.15.6:6789/0 conn(0x7f11d806d560 legacy=0x7f11d8072590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:09.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.780+0000 7f11de455640 1 -- 172.21.15.6:0/818264058 shutdown_connections 2024-03-25T19:24:09.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.780+0000 7f11de455640 1 -- 172.21.15.6:0/818264058 >> 172.21.15.6:0/818264058 conn(0x7f11d80b5cf0 msgr2=0x7f11d80b6100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:09.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.780+0000 7f11de455640 1 -- 172.21.15.6:0/818264058 shutdown_connections 2024-03-25T19:24:09.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.780+0000 7f11de455640 1 -- 172.21.15.6:0/818264058 wait complete. 2024-03-25T19:24:09.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.781+0000 7f11de455640 10 monclient: build_initial_monmap 2024-03-25T19:24:09.783 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.781+0000 7f11de455640 10 monclient: monmap: 2024-03-25T19:24:09.783 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:24:09.783 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:24:09.784 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:24:09.784 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:24:09.784 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:09.784 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:24:09.784 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:24:09.784 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:24:09.784 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:24:09.784 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:09.784 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.781+0000 7f11de455640 1 librados: starting msgr at 2024-03-25T19:24:09.784 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.781+0000 7f11de455640 1 librados: starting objecter 2024-03-25T19:24:09.784 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.781+0000 7f11de455640 1 Processor -- start 2024-03-25T19:24:09.784 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.782+0000 7f11de455640 1 -- start start 2024-03-25T19:24:09.784 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.782+0000 7f11de455640 1 librados: setting wanted keys 2024-03-25T19:24:09.784 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.782+0000 7f11de455640 1 librados: calling monclient init 2024-03-25T19:24:09.785 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.782+0000 7f11de455640 10 monclient: init 2024-03-25T19:24:09.785 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.782+0000 7f11de455640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:09.785 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.782+0000 7f11de455640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:24:09.785 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.782+0000 7f11de455640 10 monclient(hunting): picked mon.noname-a con 0x7f11d806d180 addr v1:172.21.15.6:6789/0 2024-03-25T19:24:09.785 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.782+0000 7f11de455640 10 monclient(hunting): picked mon.noname-b con 0x7f11d80b06f0 addr v1:172.21.15.192:6789/0 2024-03-25T19:24:09.785 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.782+0000 7f11de455640 10 monclient(hunting): picked mon.noname-c con 0x7f11d80b7e60 addr v1:172.21.15.6:6790/0 2024-03-25T19:24:09.785 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.782+0000 7f11de455640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d80b9fe0 con 0x7f11d806d180 2024-03-25T19:24:09.785 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.782+0000 7f11de455640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d80b8420 con 0x7f11d80b06f0 2024-03-25T19:24:09.785 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.782+0000 7f11de455640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d80b8700 con 0x7f11d80b7e60 2024-03-25T19:24:09.785 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.782+0000 7f11de455640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:09.785 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.782+0000 7f11de455640 10 monclient(hunting): authenticate will time out at 6554.022373s 2024-03-25T19:24:09.789 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.789+0000 7f11d7fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f11d806d180 0x7f11d80ba6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43018/0 (socket says 172.21.15.6:43018) 2024-03-25T19:24:09.789 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.789+0000 7f11d7fff640 1 -- 172.21.15.6:0/3244876807 learned_addr learned my addr 172.21.15.6:0/3244876807 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:24:09.795 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.795+0000 7f11d4ff9640 10 client.?.objecter ms_handle_connect 0x7f11d806d180 2024-03-25T19:24:09.795 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.795+0000 7f11d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:09.799 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.799+0000 7f11d4ff9640 10 client.?.objecter ms_handle_connect 0x7f11d80b7e60 2024-03-25T19:24:09.799 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.799+0000 7f11d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:09.802 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.802+0000 7f11d4ff9640 1 -- 172.21.15.6:0/3244876807 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 398028241 0 0) 0x7f11d80b9fe0 con 0x7f11d806d180 2024-03-25T19:24:09.802 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.802+0000 7f11d4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:24:09.802 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.802+0000 7f11d4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:09.802 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.802+0000 7f11d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:09.802 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.802+0000 7f11d4ff9640 10 monclient(hunting): my global_id is 4677 2024-03-25T19:24:09.802 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.802+0000 7f11d4ff9640 1 -- 172.21.15.6:0/3244876807 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11c4004eb0 con 0x7f11d806d180 2024-03-25T19:24:09.803 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.803+0000 7f11d4ff9640 10 client.?.objecter ms_handle_connect 0x7f11d80b06f0 2024-03-25T19:24:09.803 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.803+0000 7f11d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:09.813 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.812+0000 7f11d4ff9640 1 -- 172.21.15.6:0/3244876807 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2943753604 0 0) 0x7f11d80b8700 con 0x7f11d80b7e60 2024-03-25T19:24:09.813 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.812+0000 7f11d4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:24:09.813 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.812+0000 7f11d4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:09.813 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.812+0000 7f11d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:09.813 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.812+0000 7f11d4ff9640 10 monclient(hunting): my global_id is 4679 2024-03-25T19:24:09.813 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.812+0000 7f11d4ff9640 1 -- 172.21.15.6:0/3244876807 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11d80b9fe0 con 0x7f11d80b7e60 2024-03-25T19:24:09.813 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.813+0000 7f11d4ff9640 1 -- 172.21.15.6:0/3244876807 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 471211820 0 0) 0x7f11d80b8420 con 0x7f11d80b06f0 2024-03-25T19:24:09.813 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.813+0000 7f11d4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:24:09.813 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.813+0000 7f11d4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:09.813 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.813+0000 7f11d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:09.813 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.813+0000 7f11d4ff9640 10 monclient(hunting): my global_id is 4687 2024-03-25T19:24:09.814 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.813+0000 7f11d4ff9640 1 -- 172.21.15.6:0/3244876807 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11d80b8700 con 0x7f11d80b06f0 2024-03-25T19:24:09.829 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.829+0000 7f11d4ff9640 1 -- 172.21.15.6:0/3244876807 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 34446080 0 0) 0x7f11c4004eb0 con 0x7f11d806d180 2024-03-25T19:24:09.829 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.829+0000 7f11d4ff9640 1 -- 172.21.15.6:0/3244876807 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11d80b8420 con 0x7f11d806d180 2024-03-25T19:24:09.829 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.829+0000 7f11d4ff9640 1 -- 172.21.15.6:0/3244876807 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f11c80019b0 con 0x7f11d806d180 2024-03-25T19:24:09.829 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.829+0000 7f11d4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:09.829 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.829+0000 7f11d4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:24:09.829 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.829+0000 7f11d4ff9640 10 monclient(hunting): dump: 2024-03-25T19:24:09.829 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:09.829 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:09.829 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:09.829 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:09.829 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:09.829 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:09.830 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:09.830 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:09.830 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:09.830 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:09.830 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.829+0000 7f11d4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:24:09.834 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.834+0000 7f11d4ff9640 1 -- 172.21.15.6:0/3244876807 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3638086607 0 0) 0x7f11d80b8700 con 0x7f11d80b06f0 2024-03-25T19:24:09.834 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.834+0000 7f11d4ff9640 1 -- 172.21.15.6:0/3244876807 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11c4004eb0 con 0x7f11d80b06f0 2024-03-25T19:24:09.835 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.834+0000 7f11d4ff9640 1 -- 172.21.15.6:0/3244876807 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2786739519 0 0) 0x7f11d80b9fe0 con 0x7f11d80b7e60 2024-03-25T19:24:09.835 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.835+0000 7f11d4ff9640 1 -- 172.21.15.6:0/3244876807 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11d80b8700 con 0x7f11d80b7e60 2024-03-25T19:24:09.835 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.835+0000 7f11d4ff9640 1 -- 172.21.15.6:0/3244876807 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f11cc001b00 con 0x7f11d80b06f0 2024-03-25T19:24:09.835 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.835+0000 7f11d4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:09.835 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.835+0000 7f11d4ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:24:09.835 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.835+0000 7f11d4ff9640 10 monclient(hunting): dump: 2024-03-25T19:24:09.835 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:09.835 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:09.835 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:09.835 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:09.835 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:09.835 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:09.835 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:09.835 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:09.836 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:09.836 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:09.836 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.835+0000 7f11d4ff9640 1 -- 172.21.15.6:0/3244876807 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f11d0001a80 con 0x7f11d80b7e60 2024-03-25T19:24:09.836 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.835+0000 7f11d4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:09.836 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.835+0000 7f11d4ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:24:09.836 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.835+0000 7f11d4ff9640 10 monclient(hunting): dump: 2024-03-25T19:24:09.836 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:09.836 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:09.836 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:09.836 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:09.836 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:09.836 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:09.836 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:09.836 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:09.836 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:09.836 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:09.844 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.844+0000 7f11d4ff9640 1 -- 172.21.15.6:0/3244876807 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 694854166 0 0) 0x7f11d80b8420 con 0x7f11d806d180 2024-03-25T19:24:09.844 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.844+0000 7f11d4ff9640 1 -- 172.21.15.6:0/3244876807 >> v1:172.21.15.6:6790/0 conn(0x7f11d80b7e60 legacy=0x7f11d813f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:09.844 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.844+0000 7f11d4ff9640 1 -- 172.21.15.6:0/3244876807 >> v1:172.21.15.192:6789/0 conn(0x7f11d80b06f0 legacy=0x7f11d81db6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:09.844 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.844+0000 7f11d4ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:24:09.844 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.844+0000 7f11d4ff9640 1 monclient: found mon.a 2024-03-25T19:24:09.844 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.844+0000 7f11d4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:09.844 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.844+0000 7f11d4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:24:09.844 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.844+0000 7f11d4ff9640 1 -- 172.21.15.6:0/3244876807 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d80badf0 con 0x7f11d806d180 2024-03-25T19:24:09.845 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.844+0000 7f11d4ff9640 10 monclient: _finish_auth 0 2024-03-25T19:24:09.845 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.844+0000 7f11d4ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:24:09.845 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.844+0000 7f11d4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:09.845 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.844+0000 7f11de455640 5 monclient: authenticate success, global_id 4677 2024-03-25T19:24:09.845 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.844+0000 7f11de455640 10 monclient: _renew_subs 2024-03-25T19:24:09.845 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.844+0000 7f11de455640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:24:09.845 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.844+0000 7f11de455640 1 -- 172.21.15.6:0/3244876807 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11d81dbe00 con 0x7f11d806d180 2024-03-25T19:24:09.845 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.844+0000 7f11de455640 10 client.4677.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:24:09.845 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.844+0000 7f11de455640 10 monclient: _renew_subs 2024-03-25T19:24:09.845 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.844+0000 7f11de455640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:24:09.845 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.844+0000 7f11de455640 1 -- 172.21.15.6:0/3244876807 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f11d80b8e00 con 0x7f11d806d180 2024-03-25T19:24:09.845 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.844+0000 7f11de455640 1 librados: init done 2024-03-25T19:24:09.846 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.845+0000 7f11de455640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:09.846 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.845+0000 7f11de455640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:09.846 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.845+0000 7f11de455640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:24:09.846 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:09.845+0000 7f11de455640 1 -- 172.21.15.6:0/3244876807 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11d81dbff0 con 0x7f11d806d180 2024-03-25T19:24:10.185 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:10.185+0000 7f11d4ff9640 1 -- 172.21.15.6:0/3244876807 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11c80047e0 con 0x7f11d806d180 2024-03-25T19:24:10.185 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:10.185+0000 7f11d4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:10.185 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:10.185+0000 7f11d4ff9640 1 -- 172.21.15.6:0/3244876807 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f11c8004af0 con 0x7f11d806d180 2024-03-25T19:24:10.185 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:10.185+0000 7f11d4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:10.185 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:10.185+0000 7f11d4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:24:10.186 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:10.185+0000 7f11d4ff9640 10 monclient: dump: 2024-03-25T19:24:10.186 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:10.186 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:10.186 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:10.186 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:10.186 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:10.186 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:10.186 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:10.186 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:10.186 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:10.186 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:10.186 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:10.185+0000 7f11d67fc640 4 set_mon_vals no callback set 2024-03-25T19:24:10.186 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:10.185+0000 7f11d4ff9640 1 -- 172.21.15.6:0/3244876807 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f11c801ab00 con 0x7f11d806d180 2024-03-25T19:24:10.187 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:10.186+0000 7f11d77fe640 10 monclient: get_auth_request con 0x7f11c406e180 auth_method 0 2024-03-25T19:24:10.187 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:10.186+0000 7f11d4ff9640 10 client.4677.objecter ms_handle_connect 0x7f11c406e180 2024-03-25T19:24:10.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:10.192+0000 7f11d4ff9640 1 -- 172.21.15.6:0/3244876807 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3970010106 0 0) 0x7f11c8054850 con 0x7f11d806d180 2024-03-25T19:24:10.193 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:10.192+0000 7f11d4ff9640 10 client.4677.objecter ms_dispatch 0x7f11d8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:24:10.193 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:10.192+0000 7f11d4ff9640 3 client.4677.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:24:10.193 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:10.192+0000 7f11d4ff9640 3 client.4677.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:24:10.193 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:10.192+0000 7f11d4ff9640 20 client.4677.objecter dump_active .. 0 homeless 2024-03-25T19:24:10.193 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:10.192+0000 7f11d4ff9640 1 -- 172.21.15.6:0/3244876807 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f11c8050190 con 0x7f11d806d180 2024-03-25T19:24:10.193 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:10.192+0000 7f11d4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:10.193 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:10.192+0000 7f11d4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:24:10.442 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:10.442+0000 7f11de455640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:10.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:10.442+0000 7f11de455640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:10.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:10.442+0000 7f11de455640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:24:10.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:10.442+0000 7f11de455640 1 -- 172.21.15.6:0/3244876807 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11d804f730 con 0x7f11d806d180 2024-03-25T19:24:10.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:10.475+0000 7f11d4ff9640 1 -- 172.21.15.6:0/3244876807 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3097771541) 0x7f11c8052290 con 0x7f11d806d180 2024-03-25T19:24:10.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:10.475+0000 7f11d4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:10.476 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:10.475+0000 7f11d4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:24:10.476 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:24:10.476 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":11,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:24:09.307681+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:24:09.307681+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:24:10.478 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:10.478+0000 7f11de455640 10 librados: watch_flush enter 2024-03-25T19:24:10.478 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:10.478+0000 7f11de455640 10 librados: watch_flush exit 2024-03-25T19:24:10.479 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:10.478+0000 7f11de455640 20 client.4677.objecter shutdown clearing up homeless session... 2024-03-25T19:24:10.479 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:10.478+0000 7f11de455640 10 client.4677.objecter successfully canceled tick 2024-03-25T19:24:10.479 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:10.478+0000 7f11de455640 1 -- 172.21.15.6:0/3244876807 >> v1:172.21.15.6:6800/2936652650 conn(0x7f11c406e180 legacy=0x7f11c4070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:10.479 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:10.478+0000 7f11de455640 10 monclient: shutdown 2024-03-25T19:24:10.479 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:10.478+0000 7f11de455640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:10.479 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:10.478+0000 7f11de455640 1 -- 172.21.15.6:0/3244876807 >> v1:172.21.15.6:6789/0 conn(0x7f11d806d180 legacy=0x7f11d80ba6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:10.479 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:10.478+0000 7f11de455640 1 -- 172.21.15.6:0/3244876807 shutdown_connections 2024-03-25T19:24:10.479 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:10.478+0000 7f11de455640 1 -- 172.21.15.6:0/3244876807 >> 172.21.15.6:0/3244876807 conn(0x7f11d80b5cf0 msgr2=0x7f11d8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:10.479 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:10.478+0000 7f11de455640 1 -- 172.21.15.6:0/3244876807 shutdown_connections 2024-03-25T19:24:10.479 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:10.478+0000 7f11de455640 1 -- 172.21.15.6:0/3244876807 wait complete. 2024-03-25T19:24:10.479 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:10.478+0000 7f11de455640 1 librados: shutdown 2024-03-25T19:24:10.492 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:24:17.299 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:24:17.414 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.414+0000 7f422d89a640 10 monclient: get_monmap_and_config 2024-03-25T19:24:17.414 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.414+0000 7f422d89a640 10 monclient: build_initial_monmap 2024-03-25T19:24:17.414 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.414+0000 7f422d89a640 10 monclient: monmap: 2024-03-25T19:24:17.415 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:24:17.415 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:17.415 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:24:17.415724+0000 2024-03-25T19:24:17.415 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:24:17.415724+0000 2024-03-25T19:24:17.415 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:17.415 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:17.415 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:24:17.415 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:24:17.415 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:24:17.415 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:17.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.415+0000 7f422d89a640 1 Processor -- start 2024-03-25T19:24:17.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.415+0000 7f422d89a640 1 -- start start 2024-03-25T19:24:17.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.415+0000 7f422d89a640 10 monclient: init 2024-03-25T19:24:17.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.416+0000 7f422d89a640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:17.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.416+0000 7f422d89a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:24:17.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.416+0000 7f422d89a640 10 monclient(hunting): picked mon.noname-b con 0x7f42280b4ff0 addr v1:172.21.15.6:6790/0 2024-03-25T19:24:17.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.416+0000 7f422d89a640 10 monclient(hunting): picked mon.noname-c con 0x7f42280b4670 addr v1:172.21.15.192:6789/0 2024-03-25T19:24:17.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.416+0000 7f422d89a640 10 monclient(hunting): picked mon.noname-a con 0x7f422813c480 addr v1:172.21.15.6:6789/0 2024-03-25T19:24:17.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.416+0000 7f422d89a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42280ab0e0 con 0x7f422813c480 2024-03-25T19:24:17.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.416+0000 7f422d89a640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42280ab3c0 con 0x7f42280b4670 2024-03-25T19:24:17.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.416+0000 7f422d89a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42280a9f30 con 0x7f42280b4ff0 2024-03-25T19:24:17.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.416+0000 7f422d89a640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:17.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.416+0000 7f422d89a640 10 monclient(hunting): authenticate will time out at 6561.656287s 2024-03-25T19:24:17.425 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.425+0000 7f4226ffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f42280b4ff0 0x7f422813bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:37404/0 (socket says 172.21.15.6:37404) 2024-03-25T19:24:17.425 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.425+0000 7f4226ffd640 1 -- 172.21.15.6:0/667044286 learned_addr learned my addr 172.21.15.6:0/667044286 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:24:17.442 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.442+0000 7f42267fc640 1 -- 172.21.15.6:0/667044286 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2398881170 0 0) 0x7f42280a9f30 con 0x7f42280b4ff0 2024-03-25T19:24:17.442 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.442+0000 7f42267fc640 10 monclient(hunting): _negotiate 2024-03-25T19:24:17.442 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.442+0000 7f42267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:17.442 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.442+0000 7f42267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:17.442 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.442+0000 7f42267fc640 10 monclient(hunting): my global_id is 4682 2024-03-25T19:24:17.442 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.442+0000 7f42267fc640 1 -- 172.21.15.6:0/667044286 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4214003520 con 0x7f42280b4ff0 2024-03-25T19:24:17.459 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.459+0000 7f42267fc640 1 -- 172.21.15.6:0/667044286 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4102974055 0 0) 0x7f42280ab3c0 con 0x7f42280b4670 2024-03-25T19:24:17.459 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.459+0000 7f42267fc640 10 monclient(hunting): _negotiate 2024-03-25T19:24:17.459 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.459+0000 7f42267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:17.459 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.459+0000 7f42267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:17.459 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.459+0000 7f42267fc640 10 monclient(hunting): my global_id is 4690 2024-03-25T19:24:17.459 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.459+0000 7f42267fc640 1 -- 172.21.15.6:0/667044286 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42280a9f30 con 0x7f42280b4670 2024-03-25T19:24:17.465 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.465+0000 7f42267fc640 1 -- 172.21.15.6:0/667044286 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 262774855 0 0) 0x7f42280ab0e0 con 0x7f422813c480 2024-03-25T19:24:17.465 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.465+0000 7f42267fc640 10 monclient(hunting): _negotiate 2024-03-25T19:24:17.465 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.465+0000 7f42267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:17.465 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.465+0000 7f42267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:17.465 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.465+0000 7f42267fc640 10 monclient(hunting): my global_id is 4680 2024-03-25T19:24:17.465 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.465+0000 7f42267fc640 1 -- 172.21.15.6:0/667044286 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42280ab3c0 con 0x7f422813c480 2024-03-25T19:24:17.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.474+0000 7f42267fc640 1 -- 172.21.15.6:0/667044286 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 523884543 0 0) 0x7f42280a9f30 con 0x7f42280b4670 2024-03-25T19:24:17.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.474+0000 7f42267fc640 1 -- 172.21.15.6:0/667044286 >> v1:172.21.15.6:6790/0 conn(0x7f42280b4ff0 legacy=0x7f422813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:17.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.474+0000 7f42267fc640 1 -- 172.21.15.6:0/667044286 >> v1:172.21.15.6:6789/0 conn(0x7f422813c480 legacy=0x7f422814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:17.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.474+0000 7f42267fc640 10 monclient: _finish_hunting 0 2024-03-25T19:24:17.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.474+0000 7f42267fc640 1 monclient: found mon.noname-c 2024-03-25T19:24:17.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.474+0000 7f42267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:17.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.474+0000 7f42267fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:24:17.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.474+0000 7f42267fc640 1 -- 172.21.15.6:0/667044286 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42280a9b70 con 0x7f42280b4670 2024-03-25T19:24:17.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.474+0000 7f42267fc640 10 monclient: _finish_auth 0 2024-03-25T19:24:17.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.474+0000 7f42267fc640 10 monclient: _check_auth_tickets 2024-03-25T19:24:17.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.474+0000 7f42267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:17.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.474+0000 7f42267fc640 1 -- 172.21.15.6:0/667044286 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f4210001670 con 0x7f42280b4670 2024-03-25T19:24:17.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.474+0000 7f42267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:17.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.474+0000 7f42267fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:24:17.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.474+0000 7f42267fc640 10 monclient: dump: 2024-03-25T19:24:17.475 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:17.476 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:17.476 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:17.476 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:17.476 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:17.476 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:17.476 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:17.476 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:17.476 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:17.476 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:17.476 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.474+0000 7f422d89a640 5 monclient: authenticate success, global_id 4690 2024-03-25T19:24:17.476 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.474+0000 7f422d89a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:17.495 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.495+0000 7f42267fc640 1 -- 172.21.15.6:0/667044286 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42100017d0 con 0x7f42280b4670 2024-03-25T19:24:17.496 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.495+0000 7f42267fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:17.496 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.495+0000 7f42267fc640 1 -- 172.21.15.6:0/667044286 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f4210001be0 con 0x7f42280b4670 2024-03-25T19:24:17.496 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.495+0000 7f42267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:17.496 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.496+0000 7f42267fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:24:17.496 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.496+0000 7f42267fc640 10 monclient: dump: 2024-03-25T19:24:17.496 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:17.496 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:17.496 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:17.496 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:17.496 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:17.496 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:17.496 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:17.496 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:17.497 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:17.497 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:17.497 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.496+0000 7f422d89a640 10 monclient: get_monmap_and_config success 2024-03-25T19:24:17.497 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.496+0000 7f422d89a640 4 set_mon_vals no callback set 2024-03-25T19:24:17.498 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.496+0000 7f422d89a640 10 monclient: shutdown 2024-03-25T19:24:17.498 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.496+0000 7f422d89a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:17.498 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.496+0000 7f422d89a640 1 -- 172.21.15.6:0/667044286 >> v1:172.21.15.192:6789/0 conn(0x7f42280b4670 legacy=0x7f42281425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:17.498 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.496+0000 7f422d89a640 1 -- 172.21.15.6:0/667044286 shutdown_connections 2024-03-25T19:24:17.498 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.496+0000 7f422d89a640 1 -- 172.21.15.6:0/667044286 >> 172.21.15.6:0/667044286 conn(0x7f42280617a0 msgr2=0x7f4228061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:17.499 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.496+0000 7f422d89a640 1 -- 172.21.15.6:0/667044286 shutdown_connections 2024-03-25T19:24:17.499 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.497+0000 7f422d89a640 1 -- 172.21.15.6:0/667044286 wait complete. 2024-03-25T19:24:17.499 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.498+0000 7f422d89a640 10 monclient: build_initial_monmap 2024-03-25T19:24:17.499 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.498+0000 7f422d89a640 10 monclient: monmap: 2024-03-25T19:24:17.499 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:24:17.499 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:24:17.499 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:24:17.499 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:24:17.499 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:17.499 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:24:17.499 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:24:17.499 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:24:17.499 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:24:17.499 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:17.500 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.498+0000 7f422d89a640 1 librados: starting msgr at 2024-03-25T19:24:17.500 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.498+0000 7f422d89a640 1 librados: starting objecter 2024-03-25T19:24:17.500 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.498+0000 7f422d89a640 1 Processor -- start 2024-03-25T19:24:17.500 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.498+0000 7f422d89a640 1 -- start start 2024-03-25T19:24:17.500 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.498+0000 7f422d89a640 1 librados: setting wanted keys 2024-03-25T19:24:17.500 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.498+0000 7f422d89a640 1 librados: calling monclient init 2024-03-25T19:24:17.500 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.498+0000 7f422d89a640 10 monclient: init 2024-03-25T19:24:17.501 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.499+0000 7f422d89a640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:17.501 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.499+0000 7f422d89a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:24:17.501 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.499+0000 7f422d89a640 10 monclient(hunting): picked mon.noname-c con 0x7f42280b4670 addr v1:172.21.15.6:6790/0 2024-03-25T19:24:17.501 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.499+0000 7f422d89a640 10 monclient(hunting): picked mon.noname-b con 0x7f42280b4ff0 addr v1:172.21.15.192:6789/0 2024-03-25T19:24:17.501 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.499+0000 7f422d89a640 10 monclient(hunting): picked mon.noname-a con 0x7f422813c480 addr v1:172.21.15.6:6789/0 2024-03-25T19:24:17.501 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.499+0000 7f422d89a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4228143a50 con 0x7f422813c480 2024-03-25T19:24:17.501 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.499+0000 7f422d89a640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4228143650 con 0x7f42280b4ff0 2024-03-25T19:24:17.501 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.499+0000 7f422d89a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4228144990 con 0x7f42280b4670 2024-03-25T19:24:17.501 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.499+0000 7f422d89a640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:17.501 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.499+0000 7f422d89a640 10 monclient(hunting): authenticate will time out at 6561.739303s 2024-03-25T19:24:17.509 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.509+0000 7f42277fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f422813c480 0x7f4228145880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47708/0 (socket says 172.21.15.6:47708) 2024-03-25T19:24:17.509 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.509+0000 7f42277fe640 1 -- 172.21.15.6:0/1805900588 learned_addr learned my addr 172.21.15.6:0/1805900588 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:24:17.512 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.512+0000 7f4224ff9640 10 client.?.objecter ms_handle_connect 0x7f42280b4670 2024-03-25T19:24:17.512 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.512+0000 7f4224ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:17.512 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.512+0000 7f4224ff9640 10 client.?.objecter ms_handle_connect 0x7f422813c480 2024-03-25T19:24:17.512 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.512+0000 7f4224ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:17.514 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.514+0000 7f4224ff9640 10 client.?.objecter ms_handle_connect 0x7f42280b4ff0 2024-03-25T19:24:17.514 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.514+0000 7f4224ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:17.523 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.523+0000 7f4224ff9640 1 -- 172.21.15.6:0/1805900588 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1508561271 0 0) 0x7f4228143650 con 0x7f42280b4ff0 2024-03-25T19:24:17.523 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.523+0000 7f4224ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:24:17.523 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.523+0000 7f4224ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:17.523 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.523+0000 7f4224ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:17.524 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.523+0000 7f4224ff9640 10 monclient(hunting): my global_id is 4693 2024-03-25T19:24:17.524 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.523+0000 7f4224ff9640 1 -- 172.21.15.6:0/1805900588 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4208004eb0 con 0x7f42280b4ff0 2024-03-25T19:24:17.524 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.524+0000 7f4224ff9640 1 -- 172.21.15.6:0/1805900588 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4043612172 0 0) 0x7f4228144990 con 0x7f42280b4670 2024-03-25T19:24:17.524 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.524+0000 7f4224ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:24:17.524 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.524+0000 7f4224ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:17.524 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.524+0000 7f4224ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:17.525 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.524+0000 7f4224ff9640 10 monclient(hunting): my global_id is 4685 2024-03-25T19:24:17.525 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.524+0000 7f4224ff9640 1 -- 172.21.15.6:0/1805900588 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4228143650 con 0x7f42280b4670 2024-03-25T19:24:17.525 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.524+0000 7f4224ff9640 1 -- 172.21.15.6:0/1805900588 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3874498822 0 0) 0x7f4228143a50 con 0x7f422813c480 2024-03-25T19:24:17.525 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.524+0000 7f4224ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:24:17.525 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.524+0000 7f4224ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:17.525 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.524+0000 7f4224ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:17.525 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.524+0000 7f4224ff9640 10 monclient(hunting): my global_id is 4683 2024-03-25T19:24:17.525 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.524+0000 7f4224ff9640 1 -- 172.21.15.6:0/1805900588 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4228144990 con 0x7f422813c480 2024-03-25T19:24:17.539 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.539+0000 7f4224ff9640 1 -- 172.21.15.6:0/1805900588 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2213205727 0 0) 0x7f4208004eb0 con 0x7f42280b4ff0 2024-03-25T19:24:17.539 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.539+0000 7f4224ff9640 1 -- 172.21.15.6:0/1805900588 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4228143a50 con 0x7f42280b4ff0 2024-03-25T19:24:17.539 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.539+0000 7f4224ff9640 1 -- 172.21.15.6:0/1805900588 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f4210001ba0 con 0x7f42280b4ff0 2024-03-25T19:24:17.539 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.539+0000 7f4224ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:17.539 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.539+0000 7f4224ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:24:17.539 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.539+0000 7f4224ff9640 10 monclient(hunting): dump: 2024-03-25T19:24:17.539 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:17.540 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:17.540 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:17.540 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:17.540 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:17.540 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:17.540 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:17.540 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:17.540 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:17.540 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:17.540 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.539+0000 7f4224ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:24:17.542 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.543+0000 7f4224ff9640 1 -- 172.21.15.6:0/1805900588 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3528091349 0 0) 0x7f4228143650 con 0x7f42280b4670 2024-03-25T19:24:17.543 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.543+0000 7f4224ff9640 1 -- 172.21.15.6:0/1805900588 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4208004eb0 con 0x7f42280b4670 2024-03-25T19:24:17.543 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.543+0000 7f4224ff9640 1 -- 172.21.15.6:0/1805900588 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f4218001950 con 0x7f42280b4670 2024-03-25T19:24:17.543 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.543+0000 7f4224ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:17.543 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.543+0000 7f4224ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:24:17.543 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.543+0000 7f4224ff9640 10 monclient(hunting): dump: 2024-03-25T19:24:17.543 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:17.543 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:17.543 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:17.543 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:17.543 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:17.543 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:17.543 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:17.543 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:17.543 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:17.543 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:17.550 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.550+0000 7f4224ff9640 1 -- 172.21.15.6:0/1805900588 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1703169578 0 0) 0x7f4228144990 con 0x7f422813c480 2024-03-25T19:24:17.550 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.550+0000 7f4224ff9640 1 -- 172.21.15.6:0/1805900588 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4228143650 con 0x7f422813c480 2024-03-25T19:24:17.550 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.550+0000 7f4224ff9640 1 -- 172.21.15.6:0/1805900588 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f4220001b50 con 0x7f422813c480 2024-03-25T19:24:17.550 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.550+0000 7f4224ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:17.550 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.550+0000 7f4224ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:24:17.550 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.550+0000 7f4224ff9640 10 monclient(hunting): dump: 2024-03-25T19:24:17.550 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:17.550 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:17.550 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:17.551 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:17.551 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:17.551 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:17.551 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:17.551 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:17.551 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:17.551 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:17.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.551+0000 7f4224ff9640 1 -- 172.21.15.6:0/1805900588 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2819307181 0 0) 0x7f4228143a50 con 0x7f42280b4ff0 2024-03-25T19:24:17.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.551+0000 7f4224ff9640 1 -- 172.21.15.6:0/1805900588 >> v1:172.21.15.6:6790/0 conn(0x7f42280b4670 legacy=0x7f42281db410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:17.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.551+0000 7f4224ff9640 1 -- 172.21.15.6:0/1805900588 >> v1:172.21.15.6:6789/0 conn(0x7f422813c480 legacy=0x7f4228145880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:17.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.551+0000 7f4224ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:24:17.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.551+0000 7f4224ff9640 1 monclient: found mon.b 2024-03-25T19:24:17.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.551+0000 7f4224ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:17.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.551+0000 7f4224ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:24:17.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.551+0000 7f4224ff9640 1 -- 172.21.15.6:0/1805900588 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f422813f6a0 con 0x7f42280b4ff0 2024-03-25T19:24:17.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.551+0000 7f4224ff9640 10 monclient: _finish_auth 0 2024-03-25T19:24:17.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.551+0000 7f4224ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:24:17.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.551+0000 7f4224ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:17.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.551+0000 7f422d89a640 5 monclient: authenticate success, global_id 4693 2024-03-25T19:24:17.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.551+0000 7f422d89a640 10 monclient: _renew_subs 2024-03-25T19:24:17.552 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.551+0000 7f422d89a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:24:17.553 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.551+0000 7f422d89a640 1 -- 172.21.15.6:0/1805900588 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42281dbb20 con 0x7f42280b4ff0 2024-03-25T19:24:17.553 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.551+0000 7f422d89a640 10 client.4693.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:24:17.553 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.551+0000 7f422d89a640 10 monclient: _renew_subs 2024-03-25T19:24:17.553 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.551+0000 7f422d89a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:24:17.553 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.551+0000 7f422d89a640 1 -- 172.21.15.6:0/1805900588 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f422813ed40 con 0x7f42280b4ff0 2024-03-25T19:24:17.553 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.551+0000 7f422d89a640 1 librados: init done 2024-03-25T19:24:17.553 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.552+0000 7f422d89a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:17.553 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.552+0000 7f422d89a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:17.553 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.552+0000 7f422d89a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:24:17.554 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.552+0000 7f422d89a640 1 -- 172.21.15.6:0/1805900588 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42280688c0 con 0x7f42280b4ff0 2024-03-25T19:24:17.820 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.820+0000 7f4224ff9640 1 -- 172.21.15.6:0/1805900588 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42100038a0 con 0x7f42280b4ff0 2024-03-25T19:24:17.821 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.821+0000 7f4224ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:17.821 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.821+0000 7f4224ff9640 1 -- 172.21.15.6:0/1805900588 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f4210003bb0 con 0x7f42280b4ff0 2024-03-25T19:24:17.821 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.821+0000 7f4224ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:17.821 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.821+0000 7f4224ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:24:17.821 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.821+0000 7f4224ff9640 10 monclient: dump: 2024-03-25T19:24:17.821 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:17.821 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:17.821 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:17.821 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:17.821 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:17.821 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:17.821 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:17.821 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:17.821 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:17.821 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:17.822 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.821+0000 7f4225ffb640 4 set_mon_vals no callback set 2024-03-25T19:24:17.822 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.821+0000 7f4224ff9640 1 -- 172.21.15.6:0/1805900588 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f4210019b80 con 0x7f42280b4ff0 2024-03-25T19:24:17.822 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.822+0000 7f4226ffd640 10 monclient: get_auth_request con 0x7f420806e0b0 auth_method 0 2024-03-25T19:24:17.822 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.822+0000 7f4224ff9640 10 client.4693.objecter ms_handle_connect 0x7f420806e0b0 2024-03-25T19:24:17.828 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.828+0000 7f4224ff9640 1 -- 172.21.15.6:0/1805900588 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3970010106 0 0) 0x7f42100536e0 con 0x7f42280b4ff0 2024-03-25T19:24:17.828 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.828+0000 7f4224ff9640 10 client.4693.objecter ms_dispatch 0x7f4228000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:24:17.828 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.828+0000 7f4224ff9640 3 client.4693.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:24:17.828 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.828+0000 7f4224ff9640 3 client.4693.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:24:17.828 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.828+0000 7f4224ff9640 20 client.4693.objecter dump_active .. 0 homeless 2024-03-25T19:24:17.828 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.828+0000 7f4224ff9640 1 -- 172.21.15.6:0/1805900588 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f421004f020 con 0x7f42280b4ff0 2024-03-25T19:24:17.828 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.828+0000 7f4224ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:17.828 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:17.828+0000 7f4224ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:24:18.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:18.059+0000 7f422d89a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:18.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:18.059+0000 7f422d89a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:18.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:18.059+0000 7f422d89a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:24:18.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:18.059+0000 7f422d89a640 1 -- 172.21.15.6:0/1805900588 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f422806dbd0 con 0x7f42280b4ff0 2024-03-25T19:24:18.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:18.116+0000 7f4224ff9640 1 -- 172.21.15.6:0/1805900588 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3097771541) 0x7f4210051120 con 0x7f42280b4ff0 2024-03-25T19:24:18.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:18.116+0000 7f4224ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:18.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:18.116+0000 7f4224ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:24:18.116 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:24:18.116 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":11,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:24:09.307681+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:24:09.307681+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:24:18.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:18.118+0000 7f422d89a640 10 librados: watch_flush enter 2024-03-25T19:24:18.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:18.118+0000 7f422d89a640 10 librados: watch_flush exit 2024-03-25T19:24:18.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:18.118+0000 7f422d89a640 20 client.4693.objecter shutdown clearing up homeless session... 2024-03-25T19:24:18.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:18.118+0000 7f422d89a640 10 client.4693.objecter successfully canceled tick 2024-03-25T19:24:18.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:18.118+0000 7f422d89a640 1 -- 172.21.15.6:0/1805900588 >> v1:172.21.15.6:6800/2936652650 conn(0x7f420806e0b0 legacy=0x7f4208070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:18.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:18.118+0000 7f422d89a640 10 monclient: shutdown 2024-03-25T19:24:18.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:18.118+0000 7f422d89a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:18.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:18.118+0000 7f422d89a640 1 -- 172.21.15.6:0/1805900588 >> v1:172.21.15.192:6789/0 conn(0x7f42280b4ff0 legacy=0x7f4228145170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:18.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:18.118+0000 7f422d89a640 1 -- 172.21.15.6:0/1805900588 shutdown_connections 2024-03-25T19:24:18.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:18.119+0000 7f422d89a640 1 -- 172.21.15.6:0/1805900588 >> 172.21.15.6:0/1805900588 conn(0x7f42280617a0 msgr2=0x7f42281405c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:18.119 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:18.119+0000 7f422d89a640 1 -- 172.21.15.6:0/1805900588 shutdown_connections 2024-03-25T19:24:18.120 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:18.119+0000 7f422d89a640 1 -- 172.21.15.6:0/1805900588 wait complete. 2024-03-25T19:24:18.120 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:18.119+0000 7f422d89a640 1 librados: shutdown 2024-03-25T19:24:18.130 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:24:24.936 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:24:25.052 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.051+0000 7f5c9d672640 10 monclient: get_monmap_and_config 2024-03-25T19:24:25.053 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.051+0000 7f5c9d672640 10 monclient: build_initial_monmap 2024-03-25T19:24:25.053 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.051+0000 7f5c9d672640 10 monclient: monmap: 2024-03-25T19:24:25.053 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:24:25.053 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:25.053 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:24:25.053948+0000 2024-03-25T19:24:25.053 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:24:25.053948+0000 2024-03-25T19:24:25.053 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:25.053 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:25.053 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:24:25.053 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:24:25.053 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:24:25.053 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:25.055 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.052+0000 7f5c9d672640 1 Processor -- start 2024-03-25T19:24:25.055 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.052+0000 7f5c9d672640 1 -- start start 2024-03-25T19:24:25.055 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.052+0000 7f5c9d672640 10 monclient: init 2024-03-25T19:24:25.055 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.053+0000 7f5c9d672640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:25.055 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.053+0000 7f5c9d672640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:24:25.055 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.053+0000 7f5c9d672640 10 monclient(hunting): picked mon.noname-c con 0x7f5c980b4e70 addr v1:172.21.15.192:6789/0 2024-03-25T19:24:25.055 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.053+0000 7f5c9d672640 10 monclient(hunting): picked mon.noname-b con 0x7f5c980b44b0 addr v1:172.21.15.6:6790/0 2024-03-25T19:24:25.055 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.053+0000 7f5c9d672640 10 monclient(hunting): picked mon.noname-a con 0x7f5c980b1b00 addr v1:172.21.15.6:6789/0 2024-03-25T19:24:25.055 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.053+0000 7f5c9d672640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c980a96f0 con 0x7f5c980b1b00 2024-03-25T19:24:25.055 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.053+0000 7f5c9d672640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c980ab0e0 con 0x7f5c980b4e70 2024-03-25T19:24:25.055 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.053+0000 7f5c9d672640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c98145f70 con 0x7f5c980b44b0 2024-03-25T19:24:25.055 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.053+0000 7f5c9d672640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:25.056 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.053+0000 7f5c9d672640 10 monclient(hunting): authenticate will time out at 6569.294279s 2024-03-25T19:24:25.062 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.061+0000 7f5c977fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5c980b1b00 0x7f5c9814dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:42150/0 (socket says 172.21.15.6:42150) 2024-03-25T19:24:25.062 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.061+0000 7f5c977fe640 1 -- 172.21.15.6:0/36214980 learned_addr learned my addr 172.21.15.6:0/36214980 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:24:25.077 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.076+0000 7f5c967fc640 1 -- 172.21.15.6:0/36214980 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1541978967 0 0) 0x7f5c98145f70 con 0x7f5c980b44b0 2024-03-25T19:24:25.077 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.076+0000 7f5c967fc640 10 monclient(hunting): _negotiate 2024-03-25T19:24:25.077 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.076+0000 7f5c967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:25.077 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.076+0000 7f5c967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:25.077 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.076+0000 7f5c967fc640 10 monclient(hunting): my global_id is 4688 2024-03-25T19:24:25.077 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.076+0000 7f5c967fc640 1 -- 172.21.15.6:0/36214980 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c84003520 con 0x7f5c980b44b0 2024-03-25T19:24:25.078 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.077+0000 7f5c967fc640 1 -- 172.21.15.6:0/36214980 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2046211106 0 0) 0x7f5c980a96f0 con 0x7f5c980b1b00 2024-03-25T19:24:25.078 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.077+0000 7f5c967fc640 10 monclient(hunting): _negotiate 2024-03-25T19:24:25.078 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.077+0000 7f5c967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:25.078 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.077+0000 7f5c967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:25.078 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.077+0000 7f5c967fc640 10 monclient(hunting): my global_id is 4686 2024-03-25T19:24:25.079 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.077+0000 7f5c967fc640 1 -- 172.21.15.6:0/36214980 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c98145f70 con 0x7f5c980b1b00 2024-03-25T19:24:25.098 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.096+0000 7f5c967fc640 1 -- 172.21.15.6:0/36214980 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3345364186 0 0) 0x7f5c84003520 con 0x7f5c980b44b0 2024-03-25T19:24:25.098 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.096+0000 7f5c967fc640 1 -- 172.21.15.6:0/36214980 >> v1:172.21.15.192:6789/0 conn(0x7f5c980b4e70 legacy=0x7f5c9813f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:25.098 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.096+0000 7f5c967fc640 1 -- 172.21.15.6:0/36214980 >> v1:172.21.15.6:6789/0 conn(0x7f5c980b1b00 legacy=0x7f5c9814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:25.098 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.096+0000 7f5c967fc640 10 monclient: _finish_hunting 0 2024-03-25T19:24:25.098 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.096+0000 7f5c967fc640 1 monclient: found mon.noname-b 2024-03-25T19:24:25.098 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.096+0000 7f5c967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:25.098 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.096+0000 7f5c967fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-03-25T19:24:25.098 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.096+0000 7f5c967fc640 1 -- 172.21.15.6:0/36214980 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c980a9b80 con 0x7f5c980b44b0 2024-03-25T19:24:25.098 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.096+0000 7f5c967fc640 10 monclient: _finish_auth 0 2024-03-25T19:24:25.098 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.097+0000 7f5c967fc640 10 monclient: _check_auth_tickets 2024-03-25T19:24:25.098 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.097+0000 7f5c967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:25.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.097+0000 7f5c967fc640 1 -- 172.21.15.6:0/36214980 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f5c80001690 con 0x7f5c980b44b0 2024-03-25T19:24:25.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.097+0000 7f5c967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:25.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.097+0000 7f5c967fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:24:25.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.097+0000 7f5c967fc640 10 monclient: dump: 2024-03-25T19:24:25.099 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:25.099 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:25.099 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:25.099 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:25.099 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:25.099 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:25.099 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:25.099 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:25.099 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:25.099 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:25.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.097+0000 7f5c9d672640 5 monclient: authenticate success, global_id 4688 2024-03-25T19:24:25.099 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.097+0000 7f5c9d672640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:25.112 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.111+0000 7f5c967fc640 1 -- 172.21.15.6:0/36214980 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c800017f0 con 0x7f5c980b44b0 2024-03-25T19:24:25.112 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.111+0000 7f5c967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:25.113 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.111+0000 7f5c967fc640 1 -- 172.21.15.6:0/36214980 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f5c80001c00 con 0x7f5c980b44b0 2024-03-25T19:24:25.113 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.111+0000 7f5c9d672640 10 monclient: get_monmap_and_config success 2024-03-25T19:24:25.113 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.111+0000 7f5c9d672640 4 set_mon_vals no callback set 2024-03-25T19:24:25.113 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.111+0000 7f5c967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:25.113 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.111+0000 7f5c967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:24:25.113 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.111+0000 7f5c967fc640 10 monclient: dump: 2024-03-25T19:24:25.113 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:25.113 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:25.113 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:25.113 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:25.113 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:25.113 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:25.113 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:25.113 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:25.113 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:25.114 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:25.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.112+0000 7f5c9d672640 10 monclient: shutdown 2024-03-25T19:24:25.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.112+0000 7f5c9d672640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:25.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.112+0000 7f5c9d672640 1 -- 172.21.15.6:0/36214980 >> v1:172.21.15.6:6790/0 conn(0x7f5c980b44b0 legacy=0x7f5c98145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:25.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.112+0000 7f5c9d672640 1 -- 172.21.15.6:0/36214980 shutdown_connections 2024-03-25T19:24:25.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.112+0000 7f5c9d672640 1 -- 172.21.15.6:0/36214980 >> 172.21.15.6:0/36214980 conn(0x7f5c980617a0 msgr2=0x7f5c98061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:25.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.112+0000 7f5c9d672640 1 -- 172.21.15.6:0/36214980 shutdown_connections 2024-03-25T19:24:25.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.112+0000 7f5c9d672640 1 -- 172.21.15.6:0/36214980 wait complete. 2024-03-25T19:24:25.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.113+0000 7f5c9d672640 10 monclient: build_initial_monmap 2024-03-25T19:24:25.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.113+0000 7f5c9d672640 10 monclient: monmap: 2024-03-25T19:24:25.116 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:24:25.116 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:24:25.117 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:24:25.117 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:24:25.117 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:25.117 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:24:25.117 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:24:25.117 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:24:25.117 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:24:25.117 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:25.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.113+0000 7f5c9d672640 1 librados: starting msgr at 2024-03-25T19:24:25.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.113+0000 7f5c9d672640 1 librados: starting objecter 2024-03-25T19:24:25.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.113+0000 7f5c9d672640 1 Processor -- start 2024-03-25T19:24:25.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.114+0000 7f5c9d672640 1 -- start start 2024-03-25T19:24:25.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.114+0000 7f5c9d672640 1 librados: setting wanted keys 2024-03-25T19:24:25.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.114+0000 7f5c9d672640 1 librados: calling monclient init 2024-03-25T19:24:25.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.114+0000 7f5c9d672640 10 monclient: init 2024-03-25T19:24:25.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.114+0000 7f5c9d672640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:25.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.114+0000 7f5c9d672640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:24:25.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.114+0000 7f5c9d672640 10 monclient(hunting): picked mon.noname-b con 0x7f5c980b1b00 addr v1:172.21.15.192:6789/0 2024-03-25T19:24:25.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.114+0000 7f5c9d672640 10 monclient(hunting): picked mon.noname-c con 0x7f5c980b44b0 addr v1:172.21.15.6:6790/0 2024-03-25T19:24:25.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.114+0000 7f5c9d672640 10 monclient(hunting): picked mon.noname-a con 0x7f5c980b4e70 addr v1:172.21.15.6:6789/0 2024-03-25T19:24:25.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.114+0000 7f5c9d672640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c9813b5b0 con 0x7f5c980b4e70 2024-03-25T19:24:25.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.114+0000 7f5c9d672640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c9813c830 con 0x7f5c980b1b00 2024-03-25T19:24:25.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.114+0000 7f5c9d672640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c9813a1d0 con 0x7f5c980b44b0 2024-03-25T19:24:25.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.114+0000 7f5c9d672640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:25.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.114+0000 7f5c9d672640 10 monclient(hunting): authenticate will time out at 6569.355628s 2024-03-25T19:24:25.133 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.132+0000 7f5c96ffd640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f5c980b1b00 0x7f5c9813ac20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:60510/0 (socket says 172.21.15.6:60510) 2024-03-25T19:24:25.133 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.132+0000 7f5c96ffd640 1 -- 172.21.15.6:0/3826821204 learned_addr learned my addr 172.21.15.6:0/3826821204 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:24:25.134 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.133+0000 7f5c8ffff640 10 client.?.objecter ms_handle_connect 0x7f5c980b4e70 2024-03-25T19:24:25.134 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.133+0000 7f5c8ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:25.135 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.134+0000 7f5c8ffff640 10 client.?.objecter ms_handle_connect 0x7f5c980b1b00 2024-03-25T19:24:25.135 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.134+0000 7f5c8ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:25.140 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.140+0000 7f5c8ffff640 10 client.?.objecter ms_handle_connect 0x7f5c980b44b0 2024-03-25T19:24:25.140 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.140+0000 7f5c8ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:25.142 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.141+0000 7f5c8ffff640 1 -- 172.21.15.6:0/3826821204 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 697109483 0 0) 0x7f5c9813b5b0 con 0x7f5c980b4e70 2024-03-25T19:24:25.142 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.141+0000 7f5c8ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:24:25.142 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.141+0000 7f5c8ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:25.142 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.141+0000 7f5c8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:25.142 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.141+0000 7f5c8ffff640 10 monclient(hunting): my global_id is 4689 2024-03-25T19:24:25.142 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.141+0000 7f5c8ffff640 1 -- 172.21.15.6:0/3826821204 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c78004eb0 con 0x7f5c980b4e70 2024-03-25T19:24:25.144 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.143+0000 7f5c8ffff640 1 -- 172.21.15.6:0/3826821204 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1786401468 0 0) 0x7f5c9813c830 con 0x7f5c980b1b00 2024-03-25T19:24:25.144 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.143+0000 7f5c8ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:24:25.144 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.143+0000 7f5c8ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:25.144 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.143+0000 7f5c8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:25.144 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.143+0000 7f5c8ffff640 10 monclient(hunting): my global_id is 4699 2024-03-25T19:24:25.144 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.143+0000 7f5c8ffff640 1 -- 172.21.15.6:0/3826821204 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c9813b5b0 con 0x7f5c980b1b00 2024-03-25T19:24:25.146 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.145+0000 7f5c8ffff640 1 -- 172.21.15.6:0/3826821204 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 630605854 0 0) 0x7f5c9813a1d0 con 0x7f5c980b44b0 2024-03-25T19:24:25.146 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.145+0000 7f5c8ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:24:25.146 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.145+0000 7f5c8ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:25.146 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.145+0000 7f5c8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:25.146 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.145+0000 7f5c8ffff640 10 monclient(hunting): my global_id is 4691 2024-03-25T19:24:25.146 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.145+0000 7f5c8ffff640 1 -- 172.21.15.6:0/3826821204 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c9813c830 con 0x7f5c980b44b0 2024-03-25T19:24:25.167 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.166+0000 7f5c8ffff640 1 -- 172.21.15.6:0/3826821204 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3040291800 0 0) 0x7f5c9813c830 con 0x7f5c980b44b0 2024-03-25T19:24:25.167 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.166+0000 7f5c8ffff640 1 -- 172.21.15.6:0/3826821204 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c9813a1d0 con 0x7f5c980b44b0 2024-03-25T19:24:25.167 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.166+0000 7f5c8ffff640 1 -- 172.21.15.6:0/3826821204 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f5c80001b00 con 0x7f5c980b44b0 2024-03-25T19:24:25.167 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.166+0000 7f5c8ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:25.167 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.166+0000 7f5c8ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:24:25.167 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.166+0000 7f5c8ffff640 10 monclient(hunting): dump: 2024-03-25T19:24:25.167 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:25.168 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:25.168 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:25.168 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:25.168 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:25.168 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:25.168 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:25.168 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:25.168 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:25.168 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:25.168 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.166+0000 7f5c8ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:24:25.173 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.172+0000 7f5c8ffff640 1 -- 172.21.15.6:0/3826821204 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 201734629 0 0) 0x7f5c78004eb0 con 0x7f5c980b4e70 2024-03-25T19:24:25.173 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.172+0000 7f5c8ffff640 1 -- 172.21.15.6:0/3826821204 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c9813c830 con 0x7f5c980b4e70 2024-03-25T19:24:25.173 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.172+0000 7f5c8ffff640 1 -- 172.21.15.6:0/3826821204 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f5c90001cb0 con 0x7f5c980b4e70 2024-03-25T19:24:25.173 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.172+0000 7f5c8ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:25.173 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.172+0000 7f5c8ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:24:25.174 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.172+0000 7f5c8ffff640 10 monclient(hunting): dump: 2024-03-25T19:24:25.174 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:25.174 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:25.174 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:25.174 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:25.174 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:25.174 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:25.174 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:25.174 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:25.174 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:25.174 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:25.174 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.172+0000 7f5c8ffff640 1 -- 172.21.15.6:0/3826821204 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 994067052 0 0) 0x7f5c9813b5b0 con 0x7f5c980b1b00 2024-03-25T19:24:25.174 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.172+0000 7f5c8ffff640 1 -- 172.21.15.6:0/3826821204 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c78004eb0 con 0x7f5c980b1b00 2024-03-25T19:24:25.174 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.172+0000 7f5c8ffff640 1 -- 172.21.15.6:0/3826821204 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f5c88001850 con 0x7f5c980b1b00 2024-03-25T19:24:25.174 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.172+0000 7f5c8ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:25.175 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.172+0000 7f5c8ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:24:25.175 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.172+0000 7f5c8ffff640 10 monclient(hunting): dump: 2024-03-25T19:24:25.175 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:25.175 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:25.175 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:25.175 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:25.175 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:25.175 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:25.175 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:25.175 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:25.175 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:25.175 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:25.190 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.189+0000 7f5c8ffff640 1 -- 172.21.15.6:0/3826821204 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2496126123 0 0) 0x7f5c9813a1d0 con 0x7f5c980b44b0 2024-03-25T19:24:25.190 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.189+0000 7f5c8ffff640 1 -- 172.21.15.6:0/3826821204 >> v1:172.21.15.192:6789/0 conn(0x7f5c980b1b00 legacy=0x7f5c9813ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:25.190 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.189+0000 7f5c8ffff640 1 -- 172.21.15.6:0/3826821204 >> v1:172.21.15.6:6789/0 conn(0x7f5c980b4e70 legacy=0x7f5c981d7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:25.191 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.189+0000 7f5c8ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:24:25.191 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.189+0000 7f5c8ffff640 1 monclient: found mon.c 2024-03-25T19:24:25.191 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.189+0000 7f5c8ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:25.191 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.189+0000 7f5c8ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:24:25.191 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.189+0000 7f5c8ffff640 1 -- 172.21.15.6:0/3826821204 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c9813cc50 con 0x7f5c980b44b0 2024-03-25T19:24:25.191 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.189+0000 7f5c8ffff640 10 monclient: _finish_auth 0 2024-03-25T19:24:25.191 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.189+0000 7f5c8ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:24:25.191 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.189+0000 7f5c8ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:25.191 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.189+0000 7f5c9d672640 5 monclient: authenticate success, global_id 4691 2024-03-25T19:24:25.191 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.189+0000 7f5c9d672640 10 monclient: _renew_subs 2024-03-25T19:24:25.191 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.189+0000 7f5c9d672640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:24:25.191 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.189+0000 7f5c9d672640 1 -- 172.21.15.6:0/3826821204 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c981e30a0 con 0x7f5c980b44b0 2024-03-25T19:24:25.191 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.189+0000 7f5c9d672640 10 client.4691.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:24:25.191 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.189+0000 7f5c9d672640 10 monclient: _renew_subs 2024-03-25T19:24:25.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.189+0000 7f5c9d672640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:24:25.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.189+0000 7f5c9d672640 1 -- 172.21.15.6:0/3826821204 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c981e3410 con 0x7f5c980b44b0 2024-03-25T19:24:25.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.189+0000 7f5c9d672640 1 librados: init done 2024-03-25T19:24:25.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.190+0000 7f5c9d672640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:25.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.190+0000 7f5c9d672640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:25.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.190+0000 7f5c9d672640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:24:25.192 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.190+0000 7f5c9d672640 1 -- 172.21.15.6:0/3826821204 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c9813c6a0 con 0x7f5c980b44b0 2024-03-25T19:24:25.521 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.520+0000 7f5c8ffff640 1 -- 172.21.15.6:0/3826821204 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c800038a0 con 0x7f5c980b44b0 2024-03-25T19:24:25.521 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.520+0000 7f5c8ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:25.521 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.520+0000 7f5c8ffff640 1 -- 172.21.15.6:0/3826821204 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f5c80003bb0 con 0x7f5c980b44b0 2024-03-25T19:24:25.521 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.520+0000 7f5c8ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:25.521 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.520+0000 7f5c8ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:24:25.522 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.520+0000 7f5c8ffff640 10 monclient: dump: 2024-03-25T19:24:25.522 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:25.522 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:25.522 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:25.522 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:25.522 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:25.522 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:25.522 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:25.522 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:25.522 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:25.522 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:25.522 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.520+0000 7f5c95ffb640 4 set_mon_vals no callback set 2024-03-25T19:24:25.523 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.521+0000 7f5c8ffff640 1 -- 172.21.15.6:0/3826821204 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f5c80019b80 con 0x7f5c980b44b0 2024-03-25T19:24:25.523 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.521+0000 7f5c96ffd640 10 monclient: get_auth_request con 0x7f5c7806e0f0 auth_method 0 2024-03-25T19:24:25.523 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.522+0000 7f5c8ffff640 10 client.4691.objecter ms_handle_connect 0x7f5c7806e0f0 2024-03-25T19:24:25.528 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.527+0000 7f5c8ffff640 1 -- 172.21.15.6:0/3826821204 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3970010106 0 0) 0x7f5c800535a0 con 0x7f5c980b44b0 2024-03-25T19:24:25.528 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.527+0000 7f5c8ffff640 10 client.4691.objecter ms_dispatch 0x7f5c98000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:24:25.528 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.527+0000 7f5c8ffff640 3 client.4691.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:24:25.528 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.527+0000 7f5c8ffff640 3 client.4691.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:24:25.528 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.527+0000 7f5c8ffff640 20 client.4691.objecter dump_active .. 0 homeless 2024-03-25T19:24:25.528 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.527+0000 7f5c8ffff640 1 -- 172.21.15.6:0/3826821204 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5c8004f0b0 con 0x7f5c980b44b0 2024-03-25T19:24:25.528 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.527+0000 7f5c8ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:25.528 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.527+0000 7f5c8ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:24:25.767 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.766+0000 7f5c9d672640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:25.767 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.766+0000 7f5c9d672640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:25.767 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.766+0000 7f5c9d672640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:24:25.767 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.766+0000 7f5c9d672640 1 -- 172.21.15.6:0/3826821204 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5c9813b760 con 0x7f5c980b44b0 2024-03-25T19:24:25.811 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.811+0000 7f5c8ffff640 1 -- 172.21.15.6:0/3826821204 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3097771541) 0x7f5c9813b760 con 0x7f5c980b44b0 2024-03-25T19:24:25.812 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.811+0000 7f5c8ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:25.812 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.811+0000 7f5c8ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:24:25.812 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:24:25.812 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":11,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:24:09.307681+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:24:09.307681+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:24:25.814 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.813+0000 7f5c9d672640 10 librados: watch_flush enter 2024-03-25T19:24:25.814 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.813+0000 7f5c9d672640 10 librados: watch_flush exit 2024-03-25T19:24:25.815 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.813+0000 7f5c9d672640 20 client.4691.objecter shutdown clearing up homeless session... 2024-03-25T19:24:25.815 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.813+0000 7f5c9d672640 10 client.4691.objecter successfully canceled tick 2024-03-25T19:24:25.815 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.813+0000 7f5c9d672640 1 -- 172.21.15.6:0/3826821204 >> v1:172.21.15.6:6800/2936652650 conn(0x7f5c7806e0f0 legacy=0x7f5c780705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:25.815 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.813+0000 7f5c9d672640 10 monclient: shutdown 2024-03-25T19:24:25.815 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.813+0000 7f5c9d672640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:25.815 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.813+0000 7f5c9d672640 1 -- 172.21.15.6:0/3826821204 >> v1:172.21.15.6:6790/0 conn(0x7f5c980b44b0 legacy=0x7f5c981d7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:25.815 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.813+0000 7f5c9d672640 1 -- 172.21.15.6:0/3826821204 shutdown_connections 2024-03-25T19:24:25.815 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.813+0000 7f5c9d672640 1 -- 172.21.15.6:0/3826821204 >> 172.21.15.6:0/3826821204 conn(0x7f5c980617a0 msgr2=0x7f5c980b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:25.815 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.813+0000 7f5c9d672640 1 -- 172.21.15.6:0/3826821204 shutdown_connections 2024-03-25T19:24:25.815 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.814+0000 7f5c9d672640 1 -- 172.21.15.6:0/3826821204 wait complete. 2024-03-25T19:24:25.815 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:25.814+0000 7f5c9d672640 1 librados: shutdown 2024-03-25T19:24:25.827 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:24:29.502 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:10:46.849 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:29.513 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:10:46.860 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:29.553 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:10:46.900 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:29.558 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:10:46.876 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:29.563 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:10:46.910 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:29.568 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:10:46.886 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:29.572 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:10:46.919 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:29.591 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:10:46.909 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:29.601 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:10:46.948 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:32.633 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:24:32.752 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.752+0000 7f8e44a64640 10 monclient: get_monmap_and_config 2024-03-25T19:24:32.752 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.752+0000 7f8e44a64640 10 monclient: build_initial_monmap 2024-03-25T19:24:32.752 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.752+0000 7f8e44a64640 10 monclient: monmap: 2024-03-25T19:24:32.753 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:24:32.753 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:32.753 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:24:32.753715+0000 2024-03-25T19:24:32.753 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:24:32.753715+0000 2024-03-25T19:24:32.753 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:32.753 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:32.753 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:24:32.753 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:24:32.753 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:24:32.753 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:32.754 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.753+0000 7f8e44a64640 1 Processor -- start 2024-03-25T19:24:32.754 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.753+0000 7f8e44a64640 1 -- start start 2024-03-25T19:24:32.754 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.753+0000 7f8e44a64640 10 monclient: init 2024-03-25T19:24:32.754 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.754+0000 7f8e44a64640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:32.754 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.754+0000 7f8e44a64640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:24:32.754 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.754+0000 7f8e44a64640 10 monclient(hunting): picked mon.noname-b con 0x7f8e400b4ff0 addr v1:172.21.15.6:6790/0 2024-03-25T19:24:32.755 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.754+0000 7f8e44a64640 10 monclient(hunting): picked mon.noname-c con 0x7f8e400b4670 addr v1:172.21.15.192:6789/0 2024-03-25T19:24:32.755 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.754+0000 7f8e44a64640 10 monclient(hunting): picked mon.noname-a con 0x7f8e4013c480 addr v1:172.21.15.6:6789/0 2024-03-25T19:24:32.755 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.754+0000 7f8e44a64640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e400ab0e0 con 0x7f8e4013c480 2024-03-25T19:24:32.755 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.754+0000 7f8e44a64640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e400ab3c0 con 0x7f8e400b4670 2024-03-25T19:24:32.755 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.754+0000 7f8e44a64640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e400a9f30 con 0x7f8e400b4ff0 2024-03-25T19:24:32.755 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.754+0000 7f8e44a64640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:32.755 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.754+0000 7f8e44a64640 10 monclient(hunting): authenticate will time out at 6576.994325s 2024-03-25T19:24:32.765 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.765+0000 7f8e3dd9b640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f8e400b4670 0x7f8e401425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:45830/0 (socket says 172.21.15.6:45830) 2024-03-25T19:24:32.765 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.765+0000 7f8e3dd9b640 1 -- 172.21.15.6:0/685105442 learned_addr learned my addr 172.21.15.6:0/685105442 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:24:32.777 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.778+0000 7f8e3d59a640 1 -- 172.21.15.6:0/685105442 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4061934224 0 0) 0x7f8e400ab3c0 con 0x7f8e400b4670 2024-03-25T19:24:32.778 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.778+0000 7f8e3d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:24:32.778 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.778+0000 7f8e3d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:32.778 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.778+0000 7f8e3d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:32.778 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.778+0000 7f8e3d59a640 10 monclient(hunting): my global_id is 4702 2024-03-25T19:24:32.778 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.778+0000 7f8e3d59a640 1 -- 172.21.15.6:0/685105442 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e28003520 con 0x7f8e400b4670 2024-03-25T19:24:32.789 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.789+0000 7f8e3d59a640 1 -- 172.21.15.6:0/685105442 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1994609802 0 0) 0x7f8e400ab0e0 con 0x7f8e4013c480 2024-03-25T19:24:32.789 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.789+0000 7f8e3d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:24:32.789 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.789+0000 7f8e3d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:32.789 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.789+0000 7f8e3d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:32.789 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.789+0000 7f8e3d59a640 10 monclient(hunting): my global_id is 4692 2024-03-25T19:24:32.789 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.789+0000 7f8e3d59a640 1 -- 172.21.15.6:0/685105442 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e400ab3c0 con 0x7f8e4013c480 2024-03-25T19:24:32.794 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.795+0000 7f8e3d59a640 1 -- 172.21.15.6:0/685105442 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 506679534 0 0) 0x7f8e400a9f30 con 0x7f8e400b4ff0 2024-03-25T19:24:32.795 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.795+0000 7f8e3d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:24:32.795 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.795+0000 7f8e3d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:32.795 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.795+0000 7f8e3d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:32.795 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.795+0000 7f8e3d59a640 10 monclient(hunting): my global_id is 4694 2024-03-25T19:24:32.795 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.795+0000 7f8e3d59a640 1 -- 172.21.15.6:0/685105442 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e400ab0e0 con 0x7f8e400b4ff0 2024-03-25T19:24:32.798 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.798+0000 7f8e3d59a640 1 -- 172.21.15.6:0/685105442 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4213288022 0 0) 0x7f8e28003520 con 0x7f8e400b4670 2024-03-25T19:24:32.798 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.798+0000 7f8e3d59a640 1 -- 172.21.15.6:0/685105442 >> v1:172.21.15.6:6790/0 conn(0x7f8e400b4ff0 legacy=0x7f8e4013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:32.799 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.798+0000 7f8e3d59a640 1 -- 172.21.15.6:0/685105442 >> v1:172.21.15.6:6789/0 conn(0x7f8e4013c480 legacy=0x7f8e4014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:32.799 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.798+0000 7f8e3d59a640 10 monclient: _finish_hunting 0 2024-03-25T19:24:32.799 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.798+0000 7f8e3d59a640 1 monclient: found mon.noname-c 2024-03-25T19:24:32.799 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.798+0000 7f8e3d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:32.799 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.798+0000 7f8e3d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:24:32.799 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.798+0000 7f8e3d59a640 1 -- 172.21.15.6:0/685105442 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e400a9b70 con 0x7f8e400b4670 2024-03-25T19:24:32.799 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.798+0000 7f8e3d59a640 10 monclient: _finish_auth 0 2024-03-25T19:24:32.799 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.798+0000 7f8e3d59a640 10 monclient: _check_auth_tickets 2024-03-25T19:24:32.799 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.798+0000 7f8e3d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:32.799 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.798+0000 7f8e3d59a640 1 -- 172.21.15.6:0/685105442 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f8e34001690 con 0x7f8e400b4670 2024-03-25T19:24:32.799 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.798+0000 7f8e3d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:32.799 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.798+0000 7f8e3d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:24:32.799 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.798+0000 7f8e3d59a640 10 monclient: dump: 2024-03-25T19:24:32.799 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:32.799 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:32.800 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:32.800 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:32.800 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:32.800 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:32.800 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:32.800 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:32.800 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:32.800 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:32.800 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.798+0000 7f8e44a64640 5 monclient: authenticate success, global_id 4702 2024-03-25T19:24:32.800 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.798+0000 7f8e44a64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:32.814 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.814+0000 7f8e3d59a640 1 -- 172.21.15.6:0/685105442 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e34001870 con 0x7f8e400b4670 2024-03-25T19:24:32.814 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.814+0000 7f8e3d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:32.814 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.814+0000 7f8e3d59a640 1 -- 172.21.15.6:0/685105442 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f8e34001c80 con 0x7f8e400b4670 2024-03-25T19:24:32.814 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.814+0000 7f8e3d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:32.814 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.814+0000 7f8e3d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:24:32.814 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.814+0000 7f8e3d59a640 10 monclient: dump: 2024-03-25T19:24:32.815 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:32.815 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:32.815 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:32.815 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:32.815 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:32.815 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:32.815 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:32.815 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:32.815 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:32.815 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:32.815 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.814+0000 7f8e44a64640 10 monclient: get_monmap_and_config success 2024-03-25T19:24:32.816 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.814+0000 7f8e44a64640 4 set_mon_vals no callback set 2024-03-25T19:24:32.817 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.815+0000 7f8e44a64640 10 monclient: shutdown 2024-03-25T19:24:32.817 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.815+0000 7f8e44a64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:32.817 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.815+0000 7f8e44a64640 1 -- 172.21.15.6:0/685105442 >> v1:172.21.15.192:6789/0 conn(0x7f8e400b4670 legacy=0x7f8e401425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:32.817 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.815+0000 7f8e44a64640 1 -- 172.21.15.6:0/685105442 shutdown_connections 2024-03-25T19:24:32.817 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.815+0000 7f8e44a64640 1 -- 172.21.15.6:0/685105442 >> 172.21.15.6:0/685105442 conn(0x7f8e400617a0 msgr2=0x7f8e40061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:32.817 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.815+0000 7f8e44a64640 1 -- 172.21.15.6:0/685105442 shutdown_connections 2024-03-25T19:24:32.817 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.815+0000 7f8e44a64640 1 -- 172.21.15.6:0/685105442 wait complete. 2024-03-25T19:24:32.817 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.816+0000 7f8e44a64640 10 monclient: build_initial_monmap 2024-03-25T19:24:32.817 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.816+0000 7f8e44a64640 10 monclient: monmap: 2024-03-25T19:24:32.817 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:24:32.817 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:24:32.817 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:24:32.817 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:24:32.818 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:32.818 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:24:32.818 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:24:32.818 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:24:32.818 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:24:32.818 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:32.818 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.817+0000 7f8e44a64640 1 librados: starting msgr at 2024-03-25T19:24:32.818 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.817+0000 7f8e44a64640 1 librados: starting objecter 2024-03-25T19:24:32.818 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.817+0000 7f8e44a64640 1 Processor -- start 2024-03-25T19:24:32.818 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.817+0000 7f8e44a64640 1 -- start start 2024-03-25T19:24:32.818 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.817+0000 7f8e44a64640 1 librados: setting wanted keys 2024-03-25T19:24:32.818 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.817+0000 7f8e44a64640 1 librados: calling monclient init 2024-03-25T19:24:32.818 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.817+0000 7f8e44a64640 10 monclient: init 2024-03-25T19:24:32.819 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.817+0000 7f8e44a64640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:32.819 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.817+0000 7f8e44a64640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:24:32.819 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.817+0000 7f8e44a64640 10 monclient(hunting): picked mon.noname-c con 0x7f8e400b4670 addr v1:172.21.15.6:6790/0 2024-03-25T19:24:32.819 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.817+0000 7f8e44a64640 10 monclient(hunting): picked mon.noname-b con 0x7f8e400b4ff0 addr v1:172.21.15.192:6789/0 2024-03-25T19:24:32.819 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.817+0000 7f8e44a64640 10 monclient(hunting): picked mon.noname-a con 0x7f8e4013c480 addr v1:172.21.15.6:6789/0 2024-03-25T19:24:32.820 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.817+0000 7f8e44a64640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e400b0040 con 0x7f8e4013c480 2024-03-25T19:24:32.820 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.818+0000 7f8e44a64640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e400b0320 con 0x7f8e400b4ff0 2024-03-25T19:24:32.820 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.818+0000 7f8e44a64640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e400b0600 con 0x7f8e400b4670 2024-03-25T19:24:32.820 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.818+0000 7f8e44a64640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:32.820 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.818+0000 7f8e44a64640 10 monclient(hunting): authenticate will time out at 6577.057809s 2024-03-25T19:24:32.823 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.823+0000 7f8e3ed9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8e4013c480 0x7f8e4013a940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44144/0 (socket says 172.21.15.6:44144) 2024-03-25T19:24:32.823 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.823+0000 7f8e3ed9d640 1 -- 172.21.15.6:0/3091373054 learned_addr learned my addr 172.21.15.6:0/3091373054 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:24:32.824 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.824+0000 7f8e2effd640 10 client.?.objecter ms_handle_connect 0x7f8e4013c480 2024-03-25T19:24:32.824 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.824+0000 7f8e2effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:32.829 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.829+0000 7f8e2effd640 10 client.?.objecter ms_handle_connect 0x7f8e400b4ff0 2024-03-25T19:24:32.829 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.829+0000 7f8e2effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:32.831 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.831+0000 7f8e2effd640 1 -- 172.21.15.6:0/3091373054 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2985852352 0 0) 0x7f8e400b0040 con 0x7f8e4013c480 2024-03-25T19:24:32.831 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.831+0000 7f8e2effd640 10 monclient(hunting): _negotiate 2024-03-25T19:24:32.831 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.831+0000 7f8e2effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:32.831 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.831+0000 7f8e2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:32.831 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.831+0000 7f8e2effd640 10 monclient(hunting): my global_id is 4695 2024-03-25T19:24:32.831 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.831+0000 7f8e2effd640 1 -- 172.21.15.6:0/3091373054 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e20004eb0 con 0x7f8e4013c480 2024-03-25T19:24:32.833 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.833+0000 7f8e2effd640 10 client.?.objecter ms_handle_connect 0x7f8e400b4670 2024-03-25T19:24:32.833 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.833+0000 7f8e2effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:32.842 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.842+0000 7f8e2effd640 1 -- 172.21.15.6:0/3091373054 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2215088758 0 0) 0x7f8e400b0320 con 0x7f8e400b4ff0 2024-03-25T19:24:32.842 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.842+0000 7f8e2effd640 10 monclient(hunting): _negotiate 2024-03-25T19:24:32.843 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.842+0000 7f8e2effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:32.843 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.842+0000 7f8e2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:32.843 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.842+0000 7f8e2effd640 10 monclient(hunting): my global_id is 4705 2024-03-25T19:24:32.843 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.842+0000 7f8e2effd640 1 -- 172.21.15.6:0/3091373054 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e400b0040 con 0x7f8e400b4ff0 2024-03-25T19:24:32.843 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.843+0000 7f8e2effd640 1 -- 172.21.15.6:0/3091373054 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3094336242 0 0) 0x7f8e400b0600 con 0x7f8e400b4670 2024-03-25T19:24:32.843 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.843+0000 7f8e2effd640 10 monclient(hunting): _negotiate 2024-03-25T19:24:32.843 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.843+0000 7f8e2effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:32.843 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.843+0000 7f8e2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:32.843 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.843+0000 7f8e2effd640 10 monclient(hunting): my global_id is 4697 2024-03-25T19:24:32.844 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.843+0000 7f8e2effd640 1 -- 172.21.15.6:0/3091373054 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e400b0320 con 0x7f8e400b4670 2024-03-25T19:24:32.847 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.847+0000 7f8e2effd640 1 -- 172.21.15.6:0/3091373054 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 129216028 0 0) 0x7f8e20004eb0 con 0x7f8e4013c480 2024-03-25T19:24:32.847 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.847+0000 7f8e2effd640 1 -- 172.21.15.6:0/3091373054 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e400b0600 con 0x7f8e4013c480 2024-03-25T19:24:32.847 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.847+0000 7f8e2effd640 1 -- 172.21.15.6:0/3091373054 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f8e38001c00 con 0x7f8e4013c480 2024-03-25T19:24:32.847 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.847+0000 7f8e2effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:32.847 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.847+0000 7f8e2effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:24:32.847 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.847+0000 7f8e2effd640 10 monclient(hunting): dump: 2024-03-25T19:24:32.847 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:32.847 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:32.847 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:32.847 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:32.847 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:32.848 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:32.848 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:32.848 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:32.848 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:32.848 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:32.848 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.847+0000 7f8e2effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:24:32.856 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.855+0000 7f8e2effd640 1 -- 172.21.15.6:0/3091373054 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2484996314 0 0) 0x7f8e400b0600 con 0x7f8e4013c480 2024-03-25T19:24:32.856 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.855+0000 7f8e2effd640 1 -- 172.21.15.6:0/3091373054 >> v1:172.21.15.6:6790/0 conn(0x7f8e400b4670 legacy=0x7f8e400ae210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:32.856 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.855+0000 7f8e2effd640 1 -- 172.21.15.6:0/3091373054 >> v1:172.21.15.192:6789/0 conn(0x7f8e400b4ff0 legacy=0x7f8e4013a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:32.856 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.855+0000 7f8e2effd640 10 monclient: _finish_hunting 0 2024-03-25T19:24:32.856 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.855+0000 7f8e2effd640 1 monclient: found mon.a 2024-03-25T19:24:32.856 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.855+0000 7f8e2effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:32.856 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.855+0000 7f8e2effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:24:32.856 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.855+0000 7f8e2effd640 1 -- 172.21.15.6:0/3091373054 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e400af210 con 0x7f8e4013c480 2024-03-25T19:24:32.856 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.855+0000 7f8e2effd640 10 monclient: _finish_auth 0 2024-03-25T19:24:32.856 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.855+0000 7f8e2effd640 10 monclient: _check_auth_tickets 2024-03-25T19:24:32.856 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.855+0000 7f8e2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:32.856 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.855+0000 7f8e44a64640 5 monclient: authenticate success, global_id 4695 2024-03-25T19:24:32.856 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.855+0000 7f8e44a64640 10 monclient: _renew_subs 2024-03-25T19:24:32.857 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.855+0000 7f8e44a64640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:24:32.857 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.855+0000 7f8e44a64640 1 -- 172.21.15.6:0/3091373054 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e400aee90 con 0x7f8e4013c480 2024-03-25T19:24:32.857 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.856+0000 7f8e44a64640 10 client.4695.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:24:32.857 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.856+0000 7f8e44a64640 10 monclient: _renew_subs 2024-03-25T19:24:32.857 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.856+0000 7f8e44a64640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:24:32.857 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.856+0000 7f8e44a64640 1 -- 172.21.15.6:0/3091373054 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e401e3570 con 0x7f8e4013c480 2024-03-25T19:24:32.857 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.856+0000 7f8e44a64640 1 librados: init done 2024-03-25T19:24:32.857 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.856+0000 7f8e44a64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:32.858 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.857+0000 7f8e44a64640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:32.858 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.857+0000 7f8e44a64640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:24:32.858 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.857+0000 7f8e44a64640 1 -- 172.21.15.6:0/3091373054 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e400af080 con 0x7f8e4013c480 2024-03-25T19:24:32.864 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.864+0000 7f8e2effd640 1 -- 172.21.15.6:0/3091373054 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e380020d0 con 0x7f8e4013c480 2024-03-25T19:24:32.864 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.864+0000 7f8e2effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:32.864 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.864+0000 7f8e2effd640 1 -- 172.21.15.6:0/3091373054 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f8e38003b10 con 0x7f8e4013c480 2024-03-25T19:24:32.864 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.864+0000 7f8e2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:32.864 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.864+0000 7f8e2effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:24:32.864 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.864+0000 7f8e2effd640 10 monclient: dump: 2024-03-25T19:24:32.864 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:32.864 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:32.865 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:32.865 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:32.865 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:32.865 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:32.865 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:32.865 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:32.865 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:32.865 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:32.865 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:32.864+0000 7f8e3cd99640 4 set_mon_vals no callback set 2024-03-25T19:24:33.197 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:33.197+0000 7f8e2effd640 1 -- 172.21.15.6:0/3091373054 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f8e38003c70 con 0x7f8e4013c480 2024-03-25T19:24:33.198 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:33.198+0000 7f8e3e59c640 10 monclient: get_auth_request con 0x7f8e2006de20 auth_method 0 2024-03-25T19:24:33.198 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:33.198+0000 7f8e2effd640 10 client.4695.objecter ms_handle_connect 0x7f8e2006de20 2024-03-25T19:24:33.204 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:33.204+0000 7f8e2effd640 1 -- 172.21.15.6:0/3091373054 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3970010106 0 0) 0x7f8e38054a30 con 0x7f8e4013c480 2024-03-25T19:24:33.204 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:33.204+0000 7f8e2effd640 10 client.4695.objecter ms_dispatch 0x7f8e40000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:24:33.204 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:33.204+0000 7f8e2effd640 3 client.4695.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:24:33.204 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:33.204+0000 7f8e2effd640 3 client.4695.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:24:33.204 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:33.204+0000 7f8e2effd640 20 client.4695.objecter dump_active .. 0 homeless 2024-03-25T19:24:33.204 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:33.204+0000 7f8e2effd640 1 -- 172.21.15.6:0/3091373054 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8e38050370 con 0x7f8e4013c480 2024-03-25T19:24:33.205 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:33.204+0000 7f8e2effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:33.205 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:33.204+0000 7f8e2effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:24:33.432 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:33.432+0000 7f8e44a64640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:33.432 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:33.432+0000 7f8e44a64640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:33.432 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:33.432+0000 7f8e44a64640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:24:33.432 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:33.432+0000 7f8e44a64640 1 -- 172.21.15.6:0/3091373054 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e400af3a0 con 0x7f8e4013c480 2024-03-25T19:24:33.463 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:33.463+0000 7f8e2effd640 1 -- 172.21.15.6:0/3091373054 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3097771541) 0x7f8e38052470 con 0x7f8e4013c480 2024-03-25T19:24:33.463 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:33.463+0000 7f8e2effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:33.463 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:33.463+0000 7f8e2effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:24:33.463 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:24:33.464 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":11,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:24:09.307681+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:24:09.307681+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:24:33.466 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:33.466+0000 7f8e44a64640 10 librados: watch_flush enter 2024-03-25T19:24:33.466 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:33.466+0000 7f8e44a64640 10 librados: watch_flush exit 2024-03-25T19:24:33.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:33.466+0000 7f8e44a64640 20 client.4695.objecter shutdown clearing up homeless session... 2024-03-25T19:24:33.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:33.466+0000 7f8e44a64640 10 client.4695.objecter successfully canceled tick 2024-03-25T19:24:33.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:33.466+0000 7f8e44a64640 1 -- 172.21.15.6:0/3091373054 >> v1:172.21.15.6:6800/2936652650 conn(0x7f8e2006de20 legacy=0x7f8e200702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:33.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:33.466+0000 7f8e44a64640 10 monclient: shutdown 2024-03-25T19:24:33.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:33.466+0000 7f8e44a64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:33.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:33.466+0000 7f8e44a64640 1 -- 172.21.15.6:0/3091373054 >> v1:172.21.15.6:6789/0 conn(0x7f8e4013c480 legacy=0x7f8e4013a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:33.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:33.466+0000 7f8e44a64640 1 -- 172.21.15.6:0/3091373054 shutdown_connections 2024-03-25T19:24:33.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:33.466+0000 7f8e44a64640 1 -- 172.21.15.6:0/3091373054 >> 172.21.15.6:0/3091373054 conn(0x7f8e400617a0 msgr2=0x7f8e400b6d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:33.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:33.466+0000 7f8e44a64640 1 -- 172.21.15.6:0/3091373054 shutdown_connections 2024-03-25T19:24:33.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:33.466+0000 7f8e44a64640 1 -- 172.21.15.6:0/3091373054 wait complete. 2024-03-25T19:24:33.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:33.466+0000 7f8e44a64640 1 librados: shutdown 2024-03-25T19:24:33.480 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:24:40.287 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:24:40.404 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.404+0000 7f0c0e167640 10 monclient: get_monmap_and_config 2024-03-25T19:24:40.404 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.404+0000 7f0c0e167640 10 monclient: build_initial_monmap 2024-03-25T19:24:40.404 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.404+0000 7f0c0e167640 10 monclient: monmap: 2024-03-25T19:24:40.404 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:24:40.404 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:40.404 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:24:40.405451+0000 2024-03-25T19:24:40.404 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:24:40.405451+0000 2024-03-25T19:24:40.404 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:40.404 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:40.404 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:24:40.405 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:24:40.405 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:24:40.405 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:40.406 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.405+0000 7f0c0e167640 1 Processor -- start 2024-03-25T19:24:40.406 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.405+0000 7f0c0e167640 1 -- start start 2024-03-25T19:24:40.406 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.405+0000 7f0c0e167640 10 monclient: init 2024-03-25T19:24:40.406 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.405+0000 7f0c0e167640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:40.406 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.405+0000 7f0c0e167640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:24:40.406 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.405+0000 7f0c0e167640 10 monclient(hunting): picked mon.noname-a con 0x7f0c080b4e70 addr v1:172.21.15.6:6789/0 2024-03-25T19:24:40.406 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.405+0000 7f0c0e167640 10 monclient(hunting): picked mon.noname-c con 0x7f0c080b44b0 addr v1:172.21.15.192:6789/0 2024-03-25T19:24:40.406 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.405+0000 7f0c0e167640 10 monclient(hunting): picked mon.noname-b con 0x7f0c080b1b00 addr v1:172.21.15.6:6790/0 2024-03-25T19:24:40.406 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.405+0000 7f0c0e167640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c080a96f0 con 0x7f0c080b4e70 2024-03-25T19:24:40.406 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.405+0000 7f0c0e167640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c080ab0e0 con 0x7f0c080b44b0 2024-03-25T19:24:40.406 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.405+0000 7f0c0e167640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c08145f70 con 0x7f0c080b1b00 2024-03-25T19:24:40.406 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.405+0000 7f0c0e167640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:40.407 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.405+0000 7f0c0e167640 10 monclient(hunting): authenticate will time out at 6584.645750s 2024-03-25T19:24:40.415 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.415+0000 7f0c06ffd640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f0c080b44b0 0x7f0c08145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:50710/0 (socket says 172.21.15.6:50710) 2024-03-25T19:24:40.415 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.415+0000 7f0c06ffd640 1 -- 172.21.15.6:0/2987711095 learned_addr learned my addr 172.21.15.6:0/2987711095 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:24:40.428 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.428+0000 7f0c067fc640 1 -- 172.21.15.6:0/2987711095 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3538816930 0 0) 0x7f0c080ab0e0 con 0x7f0c080b44b0 2024-03-25T19:24:40.428 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.428+0000 7f0c067fc640 10 monclient(hunting): _negotiate 2024-03-25T19:24:40.428 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.428+0000 7f0c067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:40.428 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.428+0000 7f0c067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:40.428 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.428+0000 7f0c067fc640 10 monclient(hunting): my global_id is 4708 2024-03-25T19:24:40.428 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.428+0000 7f0c067fc640 1 -- 172.21.15.6:0/2987711095 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bf0003520 con 0x7f0c080b44b0 2024-03-25T19:24:40.439 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.439+0000 7f0c067fc640 1 -- 172.21.15.6:0/2987711095 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 124791538 0 0) 0x7f0c080a96f0 con 0x7f0c080b4e70 2024-03-25T19:24:40.439 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.439+0000 7f0c067fc640 10 monclient(hunting): _negotiate 2024-03-25T19:24:40.439 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.439+0000 7f0c067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:40.439 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.439+0000 7f0c067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:40.439 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.439+0000 7f0c067fc640 10 monclient(hunting): my global_id is 4698 2024-03-25T19:24:40.439 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.439+0000 7f0c067fc640 1 -- 172.21.15.6:0/2987711095 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c080ab0e0 con 0x7f0c080b4e70 2024-03-25T19:24:40.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.443+0000 7f0c067fc640 1 -- 172.21.15.6:0/2987711095 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1533452729 0 0) 0x7f0c08145f70 con 0x7f0c080b1b00 2024-03-25T19:24:40.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.443+0000 7f0c067fc640 10 monclient(hunting): _negotiate 2024-03-25T19:24:40.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.443+0000 7f0c067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:40.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.443+0000 7f0c067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:40.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.443+0000 7f0c067fc640 10 monclient(hunting): my global_id is 4700 2024-03-25T19:24:40.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.443+0000 7f0c067fc640 1 -- 172.21.15.6:0/2987711095 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c080a96f0 con 0x7f0c080b1b00 2024-03-25T19:24:40.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.443+0000 7f0c067fc640 1 -- 172.21.15.6:0/2987711095 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1840533009 0 0) 0x7f0bf0003520 con 0x7f0c080b44b0 2024-03-25T19:24:40.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.443+0000 7f0c067fc640 1 -- 172.21.15.6:0/2987711095 >> v1:172.21.15.6:6790/0 conn(0x7f0c080b1b00 legacy=0x7f0c0814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:40.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.443+0000 7f0c067fc640 1 -- 172.21.15.6:0/2987711095 >> v1:172.21.15.6:6789/0 conn(0x7f0c080b4e70 legacy=0x7f0c0813f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:40.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.443+0000 7f0c067fc640 10 monclient: _finish_hunting 0 2024-03-25T19:24:40.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.443+0000 7f0c067fc640 1 monclient: found mon.noname-c 2024-03-25T19:24:40.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.443+0000 7f0c067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:40.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.443+0000 7f0c067fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:24:40.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.443+0000 7f0c067fc640 1 -- 172.21.15.6:0/2987711095 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c080a9b80 con 0x7f0c080b44b0 2024-03-25T19:24:40.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.443+0000 7f0c067fc640 10 monclient: _finish_auth 0 2024-03-25T19:24:40.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.443+0000 7f0c067fc640 10 monclient: _check_auth_tickets 2024-03-25T19:24:40.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.443+0000 7f0c067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:40.445 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.443+0000 7f0c067fc640 1 -- 172.21.15.6:0/2987711095 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f0bfc001690 con 0x7f0c080b44b0 2024-03-25T19:24:40.445 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.443+0000 7f0c067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:40.445 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.443+0000 7f0c067fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:24:40.445 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.443+0000 7f0c067fc640 10 monclient: dump: 2024-03-25T19:24:40.445 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:40.445 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:40.445 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:40.445 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:40.445 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:40.445 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:40.445 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:40.445 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:40.445 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:40.445 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:40.445 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.443+0000 7f0c0e167640 5 monclient: authenticate success, global_id 4708 2024-03-25T19:24:40.445 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.443+0000 7f0c0e167640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:40.460 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.460+0000 7f0c067fc640 1 -- 172.21.15.6:0/2987711095 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0bfc0017f0 con 0x7f0c080b44b0 2024-03-25T19:24:40.460 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.460+0000 7f0c067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:40.460 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.460+0000 7f0c067fc640 1 -- 172.21.15.6:0/2987711095 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f0bfc001c00 con 0x7f0c080b44b0 2024-03-25T19:24:40.460 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.460+0000 7f0c0e167640 10 monclient: get_monmap_and_config success 2024-03-25T19:24:40.460 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.460+0000 7f0c0e167640 4 set_mon_vals no callback set 2024-03-25T19:24:40.460 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.460+0000 7f0c067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:40.460 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.460+0000 7f0c067fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:24:40.460 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.460+0000 7f0c067fc640 10 monclient: dump: 2024-03-25T19:24:40.460 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:40.460 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:40.460 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:40.461 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:40.461 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:40.461 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:40.461 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:40.461 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:40.461 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:40.461 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:40.462 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.461+0000 7f0c0e167640 10 monclient: shutdown 2024-03-25T19:24:40.462 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.461+0000 7f0c0e167640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:40.462 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.461+0000 7f0c0e167640 1 -- 172.21.15.6:0/2987711095 >> v1:172.21.15.192:6789/0 conn(0x7f0c080b44b0 legacy=0x7f0c08145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:40.462 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.461+0000 7f0c0e167640 1 -- 172.21.15.6:0/2987711095 shutdown_connections 2024-03-25T19:24:40.463 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.461+0000 7f0c0e167640 1 -- 172.21.15.6:0/2987711095 >> 172.21.15.6:0/2987711095 conn(0x7f0c080617a0 msgr2=0x7f0c08061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:40.463 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.461+0000 7f0c0e167640 1 -- 172.21.15.6:0/2987711095 shutdown_connections 2024-03-25T19:24:40.463 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.461+0000 7f0c0e167640 1 -- 172.21.15.6:0/2987711095 wait complete. 2024-03-25T19:24:40.463 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.462+0000 7f0c0e167640 10 monclient: build_initial_monmap 2024-03-25T19:24:40.463 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.462+0000 7f0c0e167640 10 monclient: monmap: 2024-03-25T19:24:40.463 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:24:40.463 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:24:40.463 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:24:40.463 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:24:40.463 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:40.463 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:24:40.463 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:24:40.463 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:24:40.463 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:24:40.463 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:40.463 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.462+0000 7f0c0e167640 1 librados: starting msgr at 2024-03-25T19:24:40.463 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.462+0000 7f0c0e167640 1 librados: starting objecter 2024-03-25T19:24:40.464 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.462+0000 7f0c0e167640 1 Processor -- start 2024-03-25T19:24:40.464 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.462+0000 7f0c0e167640 1 -- start start 2024-03-25T19:24:40.464 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.462+0000 7f0c0e167640 1 librados: setting wanted keys 2024-03-25T19:24:40.464 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.462+0000 7f0c0e167640 1 librados: calling monclient init 2024-03-25T19:24:40.464 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.462+0000 7f0c0e167640 10 monclient: init 2024-03-25T19:24:40.465 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.463+0000 7f0c0e167640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:40.465 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.463+0000 7f0c0e167640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:24:40.465 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.463+0000 7f0c0e167640 10 monclient(hunting): picked mon.noname-b con 0x7f0c080b1b00 addr v1:172.21.15.192:6789/0 2024-03-25T19:24:40.465 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.463+0000 7f0c0e167640 10 monclient(hunting): picked mon.noname-c con 0x7f0c080b44b0 addr v1:172.21.15.6:6790/0 2024-03-25T19:24:40.465 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.463+0000 7f0c0e167640 10 monclient(hunting): picked mon.noname-a con 0x7f0c080b4e70 addr v1:172.21.15.6:6789/0 2024-03-25T19:24:40.465 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.463+0000 7f0c0e167640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c08142f50 con 0x7f0c080b4e70 2024-03-25T19:24:40.465 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.463+0000 7f0c0e167640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c0813c430 con 0x7f0c080b1b00 2024-03-25T19:24:40.465 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.464+0000 7f0c0e167640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c0813c710 con 0x7f0c080b44b0 2024-03-25T19:24:40.465 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.464+0000 7f0c0e167640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:40.466 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.464+0000 7f0c0e167640 10 monclient(hunting): authenticate will time out at 6584.703793s 2024-03-25T19:24:40.488 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.488+0000 7f0c06ffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f0c080b44b0 0x7f0c0813b1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:34744/0 (socket says 172.21.15.6:34744) 2024-03-25T19:24:40.488 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.488+0000 7f0c06ffd640 1 -- 172.21.15.6:0/111540061 learned_addr learned my addr 172.21.15.6:0/111540061 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:24:40.488 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.488+0000 7f0c077fe640 1 --1- 172.21.15.6:0/111540061 >> v1:172.21.15.192:6789/0 conn(0x7f0c080b1b00 0x7f0c0813aae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:50762/0 (socket says 172.21.15.6:50762) 2024-03-25T19:24:40.489 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.489+0000 7f0beffff640 10 client.?.objecter ms_handle_connect 0x7f0c080b44b0 2024-03-25T19:24:40.489 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.489+0000 7f0beffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:40.490 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.490+0000 7f0beffff640 10 client.?.objecter ms_handle_connect 0x7f0c080b1b00 2024-03-25T19:24:40.490 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.490+0000 7f0beffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:40.493 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.493+0000 7f0beffff640 10 client.?.objecter ms_handle_connect 0x7f0c080b4e70 2024-03-25T19:24:40.493 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.493+0000 7f0beffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:40.495 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.495+0000 7f0beffff640 1 -- 172.21.15.6:0/111540061 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2058527843 0 0) 0x7f0c0813c710 con 0x7f0c080b44b0 2024-03-25T19:24:40.495 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.495+0000 7f0beffff640 10 monclient(hunting): _negotiate 2024-03-25T19:24:40.495 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.495+0000 7f0beffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:40.495 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.495+0000 7f0beffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:40.495 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.495+0000 7f0beffff640 10 monclient(hunting): my global_id is 4703 2024-03-25T19:24:40.495 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.495+0000 7f0beffff640 1 -- 172.21.15.6:0/111540061 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bf4004eb0 con 0x7f0c080b44b0 2024-03-25T19:24:40.504 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.504+0000 7f0beffff640 1 -- 172.21.15.6:0/111540061 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 281382606 0 0) 0x7f0bf4004eb0 con 0x7f0c080b44b0 2024-03-25T19:24:40.504 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.504+0000 7f0beffff640 1 -- 172.21.15.6:0/111540061 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c0813c710 con 0x7f0c080b44b0 2024-03-25T19:24:40.505 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.504+0000 7f0beffff640 1 -- 172.21.15.6:0/111540061 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f0bfc001b80 con 0x7f0c080b44b0 2024-03-25T19:24:40.505 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.504+0000 7f0beffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:40.505 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.504+0000 7f0beffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:24:40.505 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.504+0000 7f0beffff640 10 monclient(hunting): dump: 2024-03-25T19:24:40.505 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:40.505 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:40.505 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:40.505 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:40.505 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:40.505 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:40.505 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:40.505 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:40.505 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:40.505 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:40.505 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.504+0000 7f0beffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:24:40.505 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.504+0000 7f0beffff640 1 -- 172.21.15.6:0/111540061 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2395330869 0 0) 0x7f0c0813c430 con 0x7f0c080b1b00 2024-03-25T19:24:40.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.504+0000 7f0beffff640 10 monclient(hunting): _negotiate 2024-03-25T19:24:40.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.504+0000 7f0beffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:40.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.504+0000 7f0beffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:40.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.504+0000 7f0beffff640 10 monclient(hunting): my global_id is 4711 2024-03-25T19:24:40.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.504+0000 7f0beffff640 1 -- 172.21.15.6:0/111540061 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bf4004eb0 con 0x7f0c080b1b00 2024-03-25T19:24:40.514 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.513+0000 7f0beffff640 1 -- 172.21.15.6:0/111540061 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3447482728 0 0) 0x7f0c0813c710 con 0x7f0c080b44b0 2024-03-25T19:24:40.514 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.513+0000 7f0beffff640 1 -- 172.21.15.6:0/111540061 >> v1:172.21.15.192:6789/0 conn(0x7f0c080b1b00 legacy=0x7f0c0813aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:40.514 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.513+0000 7f0beffff640 1 -- 172.21.15.6:0/111540061 >> v1:172.21.15.6:6789/0 conn(0x7f0c080b4e70 legacy=0x7f0c081e25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:40.514 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.513+0000 7f0beffff640 10 monclient: _finish_hunting 0 2024-03-25T19:24:40.514 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.513+0000 7f0beffff640 1 monclient: found mon.c 2024-03-25T19:24:40.514 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.513+0000 7f0beffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:40.514 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.513+0000 7f0beffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:24:40.514 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.513+0000 7f0beffff640 1 -- 172.21.15.6:0/111540061 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c0813c9f0 con 0x7f0c080b44b0 2024-03-25T19:24:40.514 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.513+0000 7f0beffff640 10 monclient: _finish_auth 0 2024-03-25T19:24:40.514 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.513+0000 7f0beffff640 10 monclient: _check_auth_tickets 2024-03-25T19:24:40.514 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.513+0000 7f0beffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:40.514 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.513+0000 7f0c0e167640 5 monclient: authenticate success, global_id 4703 2024-03-25T19:24:40.514 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.513+0000 7f0c0e167640 10 monclient: _renew_subs 2024-03-25T19:24:40.514 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.513+0000 7f0c0e167640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:24:40.515 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.513+0000 7f0c0e167640 1 -- 172.21.15.6:0/111540061 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c081338c0 con 0x7f0c080b44b0 2024-03-25T19:24:40.515 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.513+0000 7f0c0e167640 10 client.4703.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:24:40.515 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.513+0000 7f0c0e167640 10 monclient: _renew_subs 2024-03-25T19:24:40.515 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.513+0000 7f0c0e167640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:24:40.515 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.513+0000 7f0c0e167640 1 -- 172.21.15.6:0/111540061 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c081431c0 con 0x7f0c080b44b0 2024-03-25T19:24:40.515 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.513+0000 7f0c0e167640 1 librados: init done 2024-03-25T19:24:40.515 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.514+0000 7f0c0e167640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:40.515 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.514+0000 7f0c0e167640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:40.515 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.514+0000 7f0c0e167640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:24:40.516 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.514+0000 7f0c0e167640 1 -- 172.21.15.6:0/111540061 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c080aab20 con 0x7f0c080b44b0 2024-03-25T19:24:40.828 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.829+0000 7f0beffff640 1 -- 172.21.15.6:0/111540061 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0bfc004a90 con 0x7f0c080b44b0 2024-03-25T19:24:40.829 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.829+0000 7f0beffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:40.829 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.829+0000 7f0beffff640 1 -- 172.21.15.6:0/111540061 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f0bfc004da0 con 0x7f0c080b44b0 2024-03-25T19:24:40.829 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.829+0000 7f0beffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:40.829 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.829+0000 7f0beffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:24:40.829 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.829+0000 7f0beffff640 10 monclient: dump: 2024-03-25T19:24:40.829 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:40.829 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:40.829 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:40.829 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:40.829 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:40.829 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:40.829 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:40.829 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:40.830 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:40.830 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:40.830 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.829+0000 7f0c05ffb640 4 set_mon_vals no callback set 2024-03-25T19:24:40.830 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.829+0000 7f0beffff640 1 -- 172.21.15.6:0/111540061 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f0bfc01ad70 con 0x7f0c080b44b0 2024-03-25T19:24:40.830 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.830+0000 7f0c077fe640 10 monclient: get_auth_request con 0x7f0bf406de20 auth_method 0 2024-03-25T19:24:40.830 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.830+0000 7f0beffff640 10 client.4703.objecter ms_handle_connect 0x7f0bf406de20 2024-03-25T19:24:40.837 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.837+0000 7f0beffff640 1 -- 172.21.15.6:0/111540061 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3970010106 0 0) 0x7f0bfc0548b0 con 0x7f0c080b44b0 2024-03-25T19:24:40.837 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.837+0000 7f0beffff640 10 client.4703.objecter ms_dispatch 0x7f0c08000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:24:40.837 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.837+0000 7f0beffff640 3 client.4703.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:24:40.837 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.837+0000 7f0beffff640 3 client.4703.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:24:40.837 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.837+0000 7f0beffff640 20 client.4703.objecter dump_active .. 0 homeless 2024-03-25T19:24:40.837 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.837+0000 7f0beffff640 1 -- 172.21.15.6:0/111540061 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0bfc0501f0 con 0x7f0c080b44b0 2024-03-25T19:24:40.838 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.837+0000 7f0beffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:40.838 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:40.837+0000 7f0beffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:24:41.081 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:41.081+0000 7f0c0e167640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:41.081 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:41.081+0000 7f0c0e167640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:41.081 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:41.081+0000 7f0c0e167640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:24:41.081 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:41.081+0000 7f0c0e167640 1 -- 172.21.15.6:0/111540061 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0c0804f730 con 0x7f0c080b44b0 2024-03-25T19:24:41.102 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:41.102+0000 7f0beffff640 1 -- 172.21.15.6:0/111540061 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3097771541) 0x7f0bfc0522f0 con 0x7f0c080b44b0 2024-03-25T19:24:41.102 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:41.102+0000 7f0beffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:41.102 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:41.102+0000 7f0beffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:24:41.102 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:24:41.102 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":11,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:24:09.307681+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:24:09.307681+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:24:41.104 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:41.104+0000 7f0c0e167640 10 librados: watch_flush enter 2024-03-25T19:24:41.105 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:41.104+0000 7f0c0e167640 10 librados: watch_flush exit 2024-03-25T19:24:41.105 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:41.104+0000 7f0c0e167640 20 client.4703.objecter shutdown clearing up homeless session... 2024-03-25T19:24:41.105 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:41.104+0000 7f0c0e167640 10 client.4703.objecter successfully canceled tick 2024-03-25T19:24:41.105 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:41.104+0000 7f0c0e167640 1 -- 172.21.15.6:0/111540061 >> v1:172.21.15.6:6800/2936652650 conn(0x7f0bf406de20 legacy=0x7f0bf40702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:41.105 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:41.104+0000 7f0c0e167640 10 monclient: shutdown 2024-03-25T19:24:41.105 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:41.104+0000 7f0c0e167640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:41.105 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:41.104+0000 7f0c0e167640 1 -- 172.21.15.6:0/111540061 >> v1:172.21.15.6:6790/0 conn(0x7f0c080b44b0 legacy=0x7f0c0813b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:41.105 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:41.105+0000 7f0c0e167640 1 -- 172.21.15.6:0/111540061 shutdown_connections 2024-03-25T19:24:41.105 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:41.105+0000 7f0c0e167640 1 -- 172.21.15.6:0/111540061 >> 172.21.15.6:0/111540061 conn(0x7f0c080617a0 msgr2=0x7f0c080b59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:41.105 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:41.105+0000 7f0c0e167640 1 -- 172.21.15.6:0/111540061 shutdown_connections 2024-03-25T19:24:41.106 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:41.105+0000 7f0c0e167640 1 -- 172.21.15.6:0/111540061 wait complete. 2024-03-25T19:24:41.106 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:41.105+0000 7f0c0e167640 1 librados: shutdown 2024-03-25T19:24:41.116 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:24:47.923 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:24:48.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.041+0000 7f7a495ac640 10 monclient: get_monmap_and_config 2024-03-25T19:24:48.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.041+0000 7f7a495ac640 10 monclient: build_initial_monmap 2024-03-25T19:24:48.041 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.041+0000 7f7a495ac640 10 monclient: monmap: 2024-03-25T19:24:48.041 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:24:48.041 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:48.041 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:24:48.042209+0000 2024-03-25T19:24:48.041 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:24:48.042209+0000 2024-03-25T19:24:48.041 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:48.041 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:48.041 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:24:48.041 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:24:48.041 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:24:48.041 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:48.043 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.041+0000 7f7a495ac640 1 Processor -- start 2024-03-25T19:24:48.043 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.041+0000 7f7a495ac640 1 -- start start 2024-03-25T19:24:48.043 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.041+0000 7f7a495ac640 10 monclient: init 2024-03-25T19:24:48.043 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.042+0000 7f7a495ac640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:48.043 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.042+0000 7f7a495ac640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:24:48.043 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.042+0000 7f7a495ac640 10 monclient(hunting): picked mon.noname-b con 0x7f7a440b4e70 addr v1:172.21.15.6:6790/0 2024-03-25T19:24:48.043 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.042+0000 7f7a495ac640 10 monclient(hunting): picked mon.noname-a con 0x7f7a440b44b0 addr v1:172.21.15.6:6789/0 2024-03-25T19:24:48.043 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.042+0000 7f7a495ac640 10 monclient(hunting): picked mon.noname-c con 0x7f7a440b1b00 addr v1:172.21.15.192:6789/0 2024-03-25T19:24:48.043 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.042+0000 7f7a495ac640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a440a96f0 con 0x7f7a440b44b0 2024-03-25T19:24:48.043 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.042+0000 7f7a495ac640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a440ab0e0 con 0x7f7a440b1b00 2024-03-25T19:24:48.043 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.042+0000 7f7a495ac640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a44145f70 con 0x7f7a440b4e70 2024-03-25T19:24:48.043 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.042+0000 7f7a495ac640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:48.043 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.042+0000 7f7a495ac640 10 monclient(hunting): authenticate will time out at 6592.282280s 2024-03-25T19:24:48.044 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.045+0000 7f7a42ffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f7a440b4e70 0x7f7a4413f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:39424/0 (socket says 172.21.15.6:39424) 2024-03-25T19:24:48.044 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.045+0000 7f7a42ffd640 1 -- 172.21.15.6:0/4118046178 learned_addr learned my addr 172.21.15.6:0/4118046178 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:24:48.056 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.056+0000 7f7a41ffb640 1 -- 172.21.15.6:0/4118046178 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 552166876 0 0) 0x7f7a44145f70 con 0x7f7a440b4e70 2024-03-25T19:24:48.056 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.056+0000 7f7a41ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:24:48.056 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.056+0000 7f7a41ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:48.056 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.056+0000 7f7a41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:48.056 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.056+0000 7f7a41ffb640 10 monclient(hunting): my global_id is 4706 2024-03-25T19:24:48.057 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.056+0000 7f7a41ffb640 1 -- 172.21.15.6:0/4118046178 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a2c003520 con 0x7f7a440b4e70 2024-03-25T19:24:48.064 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.064+0000 7f7a41ffb640 1 -- 172.21.15.6:0/4118046178 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3365354576 0 0) 0x7f7a440ab0e0 con 0x7f7a440b1b00 2024-03-25T19:24:48.064 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.064+0000 7f7a41ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:24:48.064 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.064+0000 7f7a41ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:48.064 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.064+0000 7f7a41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:48.064 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.064+0000 7f7a41ffb640 10 monclient(hunting): my global_id is 4714 2024-03-25T19:24:48.064 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.064+0000 7f7a41ffb640 1 -- 172.21.15.6:0/4118046178 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a44145f70 con 0x7f7a440b1b00 2024-03-25T19:24:48.070 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.070+0000 7f7a41ffb640 1 -- 172.21.15.6:0/4118046178 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1627381483 0 0) 0x7f7a2c003520 con 0x7f7a440b4e70 2024-03-25T19:24:48.071 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.070+0000 7f7a41ffb640 1 -- 172.21.15.6:0/4118046178 >> v1:172.21.15.192:6789/0 conn(0x7f7a440b1b00 legacy=0x7f7a4414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:48.071 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.070+0000 7f7a41ffb640 1 -- 172.21.15.6:0/4118046178 >> v1:172.21.15.6:6789/0 conn(0x7f7a440b44b0 legacy=0x7f7a44145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:48.071 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.070+0000 7f7a41ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:24:48.071 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.070+0000 7f7a41ffb640 1 monclient: found mon.noname-b 2024-03-25T19:24:48.071 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.070+0000 7f7a41ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:48.071 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.070+0000 7f7a41ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-03-25T19:24:48.071 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.070+0000 7f7a41ffb640 1 -- 172.21.15.6:0/4118046178 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a440a9b80 con 0x7f7a440b4e70 2024-03-25T19:24:48.071 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.070+0000 7f7a41ffb640 10 monclient: _finish_auth 0 2024-03-25T19:24:48.071 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.070+0000 7f7a41ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:24:48.071 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.070+0000 7f7a41ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:48.071 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.070+0000 7f7a41ffb640 1 -- 172.21.15.6:0/4118046178 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f7a34001690 con 0x7f7a440b4e70 2024-03-25T19:24:48.071 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.070+0000 7f7a41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:48.071 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.070+0000 7f7a41ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:24:48.072 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.070+0000 7f7a41ffb640 10 monclient: dump: 2024-03-25T19:24:48.072 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:48.072 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:48.072 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:48.072 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:48.072 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:48.072 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:48.072 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:48.072 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:48.072 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:48.072 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:48.072 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.070+0000 7f7a495ac640 5 monclient: authenticate success, global_id 4706 2024-03-25T19:24:48.072 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.070+0000 7f7a495ac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:48.077 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.077+0000 7f7a41ffb640 1 -- 172.21.15.6:0/4118046178 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a340017f0 con 0x7f7a440b4e70 2024-03-25T19:24:48.077 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.077+0000 7f7a41ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:48.077 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.077+0000 7f7a41ffb640 1 -- 172.21.15.6:0/4118046178 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f7a34001c00 con 0x7f7a440b4e70 2024-03-25T19:24:48.077 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.077+0000 7f7a495ac640 10 monclient: get_monmap_and_config success 2024-03-25T19:24:48.077 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.077+0000 7f7a495ac640 4 set_mon_vals no callback set 2024-03-25T19:24:48.077 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.077+0000 7f7a41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:48.077 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.077+0000 7f7a41ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:24:48.078 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.077+0000 7f7a41ffb640 10 monclient: dump: 2024-03-25T19:24:48.078 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:48.078 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:48.078 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:48.078 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:48.078 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:48.078 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:48.078 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:48.078 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:48.078 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:48.078 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:48.080 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.078+0000 7f7a495ac640 10 monclient: shutdown 2024-03-25T19:24:48.080 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.078+0000 7f7a495ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:48.081 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.078+0000 7f7a495ac640 1 -- 172.21.15.6:0/4118046178 >> v1:172.21.15.6:6790/0 conn(0x7f7a440b4e70 legacy=0x7f7a4413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:48.081 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.078+0000 7f7a495ac640 1 -- 172.21.15.6:0/4118046178 shutdown_connections 2024-03-25T19:24:48.081 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.078+0000 7f7a495ac640 1 -- 172.21.15.6:0/4118046178 >> 172.21.15.6:0/4118046178 conn(0x7f7a440617a0 msgr2=0x7f7a44061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:48.081 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.078+0000 7f7a495ac640 1 -- 172.21.15.6:0/4118046178 shutdown_connections 2024-03-25T19:24:48.081 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.078+0000 7f7a495ac640 1 -- 172.21.15.6:0/4118046178 wait complete. 2024-03-25T19:24:48.081 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.079+0000 7f7a495ac640 10 monclient: build_initial_monmap 2024-03-25T19:24:48.081 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.079+0000 7f7a495ac640 10 monclient: monmap: 2024-03-25T19:24:48.081 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:24:48.081 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:24:48.081 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:24:48.081 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:24:48.081 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:48.081 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:24:48.081 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:24:48.081 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:24:48.081 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:24:48.082 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:48.082 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.079+0000 7f7a495ac640 1 librados: starting msgr at 2024-03-25T19:24:48.082 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.079+0000 7f7a495ac640 1 librados: starting objecter 2024-03-25T19:24:48.082 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.079+0000 7f7a495ac640 1 Processor -- start 2024-03-25T19:24:48.082 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.079+0000 7f7a495ac640 1 -- start start 2024-03-25T19:24:48.082 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.079+0000 7f7a495ac640 1 librados: setting wanted keys 2024-03-25T19:24:48.082 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.079+0000 7f7a495ac640 1 librados: calling monclient init 2024-03-25T19:24:48.082 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.079+0000 7f7a495ac640 10 monclient: init 2024-03-25T19:24:48.082 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.080+0000 7f7a495ac640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:48.082 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.080+0000 7f7a495ac640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:24:48.082 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.080+0000 7f7a495ac640 10 monclient(hunting): picked mon.noname-b con 0x7f7a440b1b00 addr v1:172.21.15.192:6789/0 2024-03-25T19:24:48.082 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.080+0000 7f7a495ac640 10 monclient(hunting): picked mon.noname-a con 0x7f7a440b44b0 addr v1:172.21.15.6:6789/0 2024-03-25T19:24:48.082 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.080+0000 7f7a495ac640 10 monclient(hunting): picked mon.noname-c con 0x7f7a440b4e70 addr v1:172.21.15.6:6790/0 2024-03-25T19:24:48.083 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.080+0000 7f7a495ac640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a4413c590 con 0x7f7a440b44b0 2024-03-25T19:24:48.083 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.080+0000 7f7a495ac640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a441dbd60 con 0x7f7a440b1b00 2024-03-25T19:24:48.083 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.080+0000 7f7a495ac640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a4413c870 con 0x7f7a440b4e70 2024-03-25T19:24:48.083 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.080+0000 7f7a495ac640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:48.083 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.080+0000 7f7a495ac640 10 monclient(hunting): authenticate will time out at 6592.320126s 2024-03-25T19:24:48.087 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.088+0000 7f7a437fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f7a440b4e70 0x7f7a441db650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:39462/0 (socket says 172.21.15.6:39462) 2024-03-25T19:24:48.087 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.088+0000 7f7a437fe640 1 -- 172.21.15.6:0/159882927 learned_addr learned my addr 172.21.15.6:0/159882927 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:24:48.088 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.088+0000 7f7a437fe640 1 -- 172.21.15.6:0/159882927 >> v1:172.21.15.6:6790/0 conn(0x7f7a440b4e70 legacy=0x7f7a441db650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-25T19:24:48.088 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.088+0000 7f7a437fe640 1 -- 172.21.15.6:0/159882927 >> v1:172.21.15.6:6790/0 conn(0x7f7a440b4e70 legacy=0x7f7a441db650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:24:48.089 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.089+0000 7f7a2b7fe640 10 client.?.objecter ms_handle_connect 0x7f7a440b44b0 2024-03-25T19:24:48.089 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.089+0000 7f7a2b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:48.089 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.089+0000 7f7a437fe640 1 --1- 172.21.15.6:0/159882927 >> v1:172.21.15.6:6790/0 conn(0x7f7a440b4e70 0x7f7a441db650 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-03-25T19:24:48.089 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.089+0000 7f7a437fe640 1 --1- 172.21.15.6:0/159882927 >> v1:172.21.15.6:6790/0 conn(0x7f7a440b4e70 0x7f7a441db650 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-03-25T19:24:48.089 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.089+0000 7f7a2b7fe640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.6:6790/0 2024-03-25T19:24:48.096 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.097+0000 7f7a2b7fe640 1 -- 172.21.15.6:0/159882927 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4150887919 0 0) 0x7f7a4413c590 con 0x7f7a440b44b0 2024-03-25T19:24:48.097 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.097+0000 7f7a2b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:24:48.097 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.097+0000 7f7a2b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:48.097 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.097+0000 7f7a2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:48.097 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.097+0000 7f7a2b7fe640 10 monclient(hunting): my global_id is 4707 2024-03-25T19:24:48.097 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.097+0000 7f7a2b7fe640 1 -- 172.21.15.6:0/159882927 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a30004eb0 con 0x7f7a440b44b0 2024-03-25T19:24:48.102 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.102+0000 7f7a2b7fe640 10 client.?.objecter ms_handle_connect 0x7f7a440b1b00 2024-03-25T19:24:48.102 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.102+0000 7f7a2b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:48.126 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.126+0000 7f7a2b7fe640 1 -- 172.21.15.6:0/159882927 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2432565958 0 0) 0x7f7a441dbd60 con 0x7f7a440b1b00 2024-03-25T19:24:48.126 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.126+0000 7f7a2b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:24:48.126 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.126+0000 7f7a2b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:48.126 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.126+0000 7f7a2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:48.126 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.126+0000 7f7a2b7fe640 10 monclient(hunting): my global_id is 4717 2024-03-25T19:24:48.126 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.126+0000 7f7a2b7fe640 1 -- 172.21.15.6:0/159882927 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a4413c590 con 0x7f7a440b1b00 2024-03-25T19:24:48.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.129+0000 7f7a2b7fe640 1 -- 172.21.15.6:0/159882927 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 670630089 0 0) 0x7f7a30004eb0 con 0x7f7a440b44b0 2024-03-25T19:24:48.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.129+0000 7f7a2b7fe640 1 -- 172.21.15.6:0/159882927 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a441dbd60 con 0x7f7a440b44b0 2024-03-25T19:24:48.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.129+0000 7f7a2b7fe640 1 -- 172.21.15.6:0/159882927 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f7a38001930 con 0x7f7a440b44b0 2024-03-25T19:24:48.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.129+0000 7f7a2b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:48.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.129+0000 7f7a2b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:24:48.129 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.129+0000 7f7a2b7fe640 10 monclient(hunting): dump: 2024-03-25T19:24:48.129 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:48.129 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:48.129 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:48.129 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:48.130 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:48.130 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:48.130 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:48.130 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:48.130 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:48.130 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:48.130 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.129+0000 7f7a2b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:24:48.146 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.146+0000 7f7a2b7fe640 1 -- 172.21.15.6:0/159882927 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3564568919 0 0) 0x7f7a4413c590 con 0x7f7a440b1b00 2024-03-25T19:24:48.147 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.146+0000 7f7a2b7fe640 1 -- 172.21.15.6:0/159882927 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a30004eb0 con 0x7f7a440b1b00 2024-03-25T19:24:48.147 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.146+0000 7f7a2b7fe640 1 -- 172.21.15.6:0/159882927 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f7a34001b00 con 0x7f7a440b1b00 2024-03-25T19:24:48.147 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.146+0000 7f7a2b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:48.147 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.146+0000 7f7a2b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:24:48.147 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.146+0000 7f7a2b7fe640 10 monclient(hunting): dump: 2024-03-25T19:24:48.147 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:48.147 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:48.147 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:48.147 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:48.147 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:48.147 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:48.147 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:48.147 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:48.147 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:48.148 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:48.149 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.148+0000 7f7a2b7fe640 1 -- 172.21.15.6:0/159882927 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3667102615 0 0) 0x7f7a441dbd60 con 0x7f7a440b44b0 2024-03-25T19:24:48.149 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.148+0000 7f7a2b7fe640 1 -- 172.21.15.6:0/159882927 >> v1:172.21.15.6:6790/0 conn(0x7f7a440b4e70 legacy=0x7f7a441db650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:24:48.149 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.148+0000 7f7a2b7fe640 1 -- 172.21.15.6:0/159882927 >> v1:172.21.15.192:6789/0 conn(0x7f7a440b1b00 legacy=0x7f7a441daf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:48.149 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.148+0000 7f7a2b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:24:48.149 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.148+0000 7f7a2b7fe640 1 monclient: found mon.a 2024-03-25T19:24:48.149 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.148+0000 7f7a2b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:48.149 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.148+0000 7f7a2b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:24:48.149 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.148+0000 7f7a2b7fe640 1 -- 172.21.15.6:0/159882927 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a44140160 con 0x7f7a440b44b0 2024-03-25T19:24:48.149 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.148+0000 7f7a2b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:24:48.150 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.148+0000 7f7a2b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:24:48.150 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.148+0000 7f7a2b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:48.150 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.148+0000 7f7a495ac640 5 monclient: authenticate success, global_id 4707 2024-03-25T19:24:48.150 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.148+0000 7f7a495ac640 10 monclient: _renew_subs 2024-03-25T19:24:48.150 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.148+0000 7f7a495ac640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:24:48.150 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.148+0000 7f7a495ac640 1 -- 172.21.15.6:0/159882927 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a4413af90 con 0x7f7a440b44b0 2024-03-25T19:24:48.150 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.148+0000 7f7a495ac640 10 client.4707.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:24:48.150 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.148+0000 7f7a495ac640 10 monclient: _renew_subs 2024-03-25T19:24:48.150 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.148+0000 7f7a495ac640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:24:48.150 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.148+0000 7f7a495ac640 1 -- 172.21.15.6:0/159882927 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a44133900 con 0x7f7a440b44b0 2024-03-25T19:24:48.150 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.148+0000 7f7a495ac640 1 librados: init done 2024-03-25T19:24:48.150 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.149+0000 7f7a495ac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:48.151 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.149+0000 7f7a495ac640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:48.151 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.149+0000 7f7a495ac640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:24:48.151 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.149+0000 7f7a495ac640 1 -- 172.21.15.6:0/159882927 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a4413b9f0 con 0x7f7a440b44b0 2024-03-25T19:24:48.466 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.466+0000 7f7a2b7fe640 1 -- 172.21.15.6:0/159882927 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a380035f0 con 0x7f7a440b44b0 2024-03-25T19:24:48.466 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.466+0000 7f7a2b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:48.466 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.466+0000 7f7a2b7fe640 1 -- 172.21.15.6:0/159882927 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f7a38003980 con 0x7f7a440b44b0 2024-03-25T19:24:48.466 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.466+0000 7f7a2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:48.466 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.466+0000 7f7a2b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:24:48.466 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.466+0000 7f7a2b7fe640 10 monclient: dump: 2024-03-25T19:24:48.466 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:48.466 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:48.466 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:48.466 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:48.467 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:48.467 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:48.467 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:48.467 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:48.467 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:48.467 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:48.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.466+0000 7f7a417fa640 4 set_mon_vals no callback set 2024-03-25T19:24:48.467 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.466+0000 7f7a2b7fe640 1 -- 172.21.15.6:0/159882927 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f7a38003ba0 con 0x7f7a440b44b0 2024-03-25T19:24:48.468 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.467+0000 7f7a42ffd640 10 monclient: get_auth_request con 0x7f7a3006de90 auth_method 0 2024-03-25T19:24:48.468 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.467+0000 7f7a2b7fe640 10 client.4707.objecter ms_handle_connect 0x7f7a3006de90 2024-03-25T19:24:48.473 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.473+0000 7f7a2b7fe640 1 -- 172.21.15.6:0/159882927 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3970010106 0 0) 0x7f7a380535b0 con 0x7f7a440b44b0 2024-03-25T19:24:48.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.473+0000 7f7a2b7fe640 10 client.4707.objecter ms_dispatch 0x7f7a44000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:24:48.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.473+0000 7f7a2b7fe640 3 client.4707.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:24:48.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.473+0000 7f7a2b7fe640 3 client.4707.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:24:48.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.473+0000 7f7a2b7fe640 20 client.4707.objecter dump_active .. 0 homeless 2024-03-25T19:24:48.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.473+0000 7f7a2b7fe640 1 -- 172.21.15.6:0/159882927 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7a3804f0c0 con 0x7f7a440b44b0 2024-03-25T19:24:48.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.473+0000 7f7a2b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:48.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.474+0000 7f7a2b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:24:48.692 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.692+0000 7f7a495ac640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:48.692 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.692+0000 7f7a495ac640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:48.692 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.692+0000 7f7a495ac640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-03-25T19:24:48.692 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.692+0000 7f7a495ac640 1 -- 172.21.15.6:0/159882927 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7a4404f730 con 0x7f7a440b44b0 2024-03-25T19:24:48.718 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.718+0000 7f7a2b7fe640 1 -- 172.21.15.6:0/159882927 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3097771541) 0x7f7a38050ff0 con 0x7f7a440b44b0 2024-03-25T19:24:48.718 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.718+0000 7f7a2b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:48.718 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.718+0000 7f7a2b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:24:48.718 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:24:48.718 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":11,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:24:09.307681+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:24:09.307681+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:24:48.720 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.720+0000 7f7a495ac640 10 librados: watch_flush enter 2024-03-25T19:24:48.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.720+0000 7f7a495ac640 10 librados: watch_flush exit 2024-03-25T19:24:48.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.720+0000 7f7a495ac640 20 client.4707.objecter shutdown clearing up homeless session... 2024-03-25T19:24:48.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.720+0000 7f7a495ac640 10 client.4707.objecter successfully canceled tick 2024-03-25T19:24:48.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.720+0000 7f7a495ac640 1 -- 172.21.15.6:0/159882927 >> v1:172.21.15.6:6800/2936652650 conn(0x7f7a3006de90 legacy=0x7f7a30070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:48.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.720+0000 7f7a495ac640 10 monclient: shutdown 2024-03-25T19:24:48.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.720+0000 7f7a495ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:48.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.720+0000 7f7a495ac640 1 -- 172.21.15.6:0/159882927 >> v1:172.21.15.6:6789/0 conn(0x7f7a440b44b0 legacy=0x7f7a44144cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:48.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.720+0000 7f7a495ac640 1 -- 172.21.15.6:0/159882927 shutdown_connections 2024-03-25T19:24:48.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.720+0000 7f7a495ac640 1 -- 172.21.15.6:0/159882927 >> 172.21.15.6:0/159882927 conn(0x7f7a440617a0 msgr2=0x7f7a440b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:48.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.720+0000 7f7a495ac640 1 -- 172.21.15.6:0/159882927 shutdown_connections 2024-03-25T19:24:48.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.720+0000 7f7a495ac640 1 -- 172.21.15.6:0/159882927 wait complete. 2024-03-25T19:24:48.721 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:48.720+0000 7f7a495ac640 1 librados: shutdown 2024-03-25T19:24:48.733 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:24:49.496 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:11:06.843 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:49.499 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:11:06.846 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:49.540 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:11:06.858 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:49.542 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:11:06.860 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:49.552 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:11:06.870 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:49.587 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:11:06.933 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:49.623 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:11:06.970 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:49.634 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:11:06.981 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:49.660 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:11:07.007 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:55.539 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:24:55.656 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.656+0000 7f0fa0a33640 10 monclient: get_monmap_and_config 2024-03-25T19:24:55.656 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.656+0000 7f0fa0a33640 10 monclient: build_initial_monmap 2024-03-25T19:24:55.656 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.656+0000 7f0fa0a33640 10 monclient: monmap: 2024-03-25T19:24:55.657 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:24:55.657 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:55.657 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:24:55.657855+0000 2024-03-25T19:24:55.657 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:24:55.657855+0000 2024-03-25T19:24:55.657 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:55.657 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:55.657 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:24:55.657 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:24:55.657 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:24:55.657 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:55.658 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.657+0000 7f0fa0a33640 1 Processor -- start 2024-03-25T19:24:55.658 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.657+0000 7f0fa0a33640 1 -- start start 2024-03-25T19:24:55.658 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.657+0000 7f0fa0a33640 10 monclient: init 2024-03-25T19:24:55.659 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.658+0000 7f0fa0a33640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:55.659 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.658+0000 7f0fa0a33640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:24:55.659 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.658+0000 7f0fa0a33640 10 monclient(hunting): picked mon.noname-c con 0x7f0f9c0b4e70 addr v1:172.21.15.192:6789/0 2024-03-25T19:24:55.659 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.659+0000 7f0fa0a33640 10 monclient(hunting): picked mon.noname-b con 0x7f0f9c0b44b0 addr v1:172.21.15.6:6790/0 2024-03-25T19:24:55.659 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.659+0000 7f0fa0a33640 10 monclient(hunting): picked mon.noname-a con 0x7f0f9c0b1b00 addr v1:172.21.15.6:6789/0 2024-03-25T19:24:55.659 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.659+0000 7f0fa0a33640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f9c0a96f0 con 0x7f0f9c0b1b00 2024-03-25T19:24:55.660 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.659+0000 7f0fa0a33640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f9c0ab0e0 con 0x7f0f9c0b4e70 2024-03-25T19:24:55.660 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.659+0000 7f0fa0a33640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f9c145f70 con 0x7f0f9c0b44b0 2024-03-25T19:24:55.660 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.659+0000 7f0fa0a33640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:55.660 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.659+0000 7f0fa0a33640 10 monclient(hunting): authenticate will time out at 6599.898960s 2024-03-25T19:24:55.661 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.662+0000 7f0f9ad9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0f9c0b1b00 0x7f0f9c14dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47004/0 (socket says 172.21.15.6:47004) 2024-03-25T19:24:55.661 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.662+0000 7f0f9ad9d640 1 -- 172.21.15.6:0/383774700 learned_addr learned my addr 172.21.15.6:0/383774700 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:24:55.673 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.674+0000 7f0f99d9b640 1 -- 172.21.15.6:0/383774700 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3944990816 0 0) 0x7f0f9c0a96f0 con 0x7f0f9c0b1b00 2024-03-25T19:24:55.674 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.674+0000 7f0f99d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:24:55.674 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.674+0000 7f0f99d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:55.674 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.674+0000 7f0f99d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:55.674 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.674+0000 7f0f99d9b640 10 monclient(hunting): my global_id is 4710 2024-03-25T19:24:55.674 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.674+0000 7f0f99d9b640 1 -- 172.21.15.6:0/383774700 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f88003520 con 0x7f0f9c0b1b00 2024-03-25T19:24:55.682 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.682+0000 7f0f99d9b640 1 -- 172.21.15.6:0/383774700 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1020353451 0 0) 0x7f0f9c145f70 con 0x7f0f9c0b44b0 2024-03-25T19:24:55.682 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.682+0000 7f0f99d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:24:55.682 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.682+0000 7f0f99d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:55.682 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.682+0000 7f0f99d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:55.682 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.682+0000 7f0f99d9b640 10 monclient(hunting): my global_id is 4709 2024-03-25T19:24:55.682 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.682+0000 7f0f99d9b640 1 -- 172.21.15.6:0/383774700 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f9c0a96f0 con 0x7f0f9c0b44b0 2024-03-25T19:24:55.684 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.684+0000 7f0f99d9b640 1 -- 172.21.15.6:0/383774700 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1398079882 0 0) 0x7f0f9c0ab0e0 con 0x7f0f9c0b4e70 2024-03-25T19:24:55.684 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.684+0000 7f0f99d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:24:55.684 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.684+0000 7f0f99d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:55.684 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.684+0000 7f0f99d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:55.684 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.684+0000 7f0f99d9b640 10 monclient(hunting): my global_id is 4720 2024-03-25T19:24:55.684 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.684+0000 7f0f99d9b640 1 -- 172.21.15.6:0/383774700 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f9c145f70 con 0x7f0f9c0b4e70 2024-03-25T19:24:55.692 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.692+0000 7f0f99d9b640 1 -- 172.21.15.6:0/383774700 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2859915685 0 0) 0x7f0f88003520 con 0x7f0f9c0b1b00 2024-03-25T19:24:55.692 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.692+0000 7f0f99d9b640 1 -- 172.21.15.6:0/383774700 >> v1:172.21.15.6:6790/0 conn(0x7f0f9c0b44b0 legacy=0x7f0f9c145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:55.692 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.692+0000 7f0f99d9b640 1 -- 172.21.15.6:0/383774700 >> v1:172.21.15.192:6789/0 conn(0x7f0f9c0b4e70 legacy=0x7f0f9c13f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:55.692 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.692+0000 7f0f99d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:24:55.692 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.692+0000 7f0f99d9b640 1 monclient: found mon.noname-a 2024-03-25T19:24:55.692 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.692+0000 7f0f99d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:55.692 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.692+0000 7f0f99d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-03-25T19:24:55.692 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.692+0000 7f0f99d9b640 1 -- 172.21.15.6:0/383774700 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f9c0a9b80 con 0x7f0f9c0b1b00 2024-03-25T19:24:55.693 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.692+0000 7f0f99d9b640 10 monclient: _finish_auth 0 2024-03-25T19:24:55.693 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.692+0000 7f0f99d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:24:55.693 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.692+0000 7f0f99d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:55.693 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.692+0000 7f0f99d9b640 1 -- 172.21.15.6:0/383774700 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f0f94001a10 con 0x7f0f9c0b1b00 2024-03-25T19:24:55.693 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.692+0000 7f0f99d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:55.693 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.692+0000 7f0f99d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:24:55.693 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.692+0000 7f0f99d9b640 10 monclient: dump: 2024-03-25T19:24:55.693 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:55.693 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:55.693 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:55.693 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:55.693 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:55.693 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:55.694 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:55.694 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:55.694 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:55.694 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:55.694 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.692+0000 7f0fa0a33640 5 monclient: authenticate success, global_id 4710 2024-03-25T19:24:55.694 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.692+0000 7f0fa0a33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:55.703 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.703+0000 7f0f99d9b640 1 -- 172.21.15.6:0/383774700 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f94001b70 con 0x7f0f9c0b1b00 2024-03-25T19:24:55.703 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.703+0000 7f0f99d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:55.703 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.703+0000 7f0f99d9b640 1 -- 172.21.15.6:0/383774700 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f0f94001f80 con 0x7f0f9c0b1b00 2024-03-25T19:24:55.703 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.703+0000 7f0fa0a33640 10 monclient: get_monmap_and_config success 2024-03-25T19:24:55.703 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.703+0000 7f0fa0a33640 4 set_mon_vals no callback set 2024-03-25T19:24:55.703 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.703+0000 7f0f99d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:55.704 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.703+0000 7f0f99d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:24:55.704 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.703+0000 7f0f99d9b640 10 monclient: dump: 2024-03-25T19:24:55.704 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:55.704 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:55.704 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:55.704 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:55.705 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:55.705 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:55.705 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:55.705 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:55.705 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:55.705 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:55.707 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.704+0000 7f0fa0a33640 10 monclient: shutdown 2024-03-25T19:24:55.707 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.704+0000 7f0fa0a33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:55.707 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.704+0000 7f0fa0a33640 1 -- 172.21.15.6:0/383774700 >> v1:172.21.15.6:6789/0 conn(0x7f0f9c0b1b00 legacy=0x7f0f9c14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:55.707 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.704+0000 7f0fa0a33640 1 -- 172.21.15.6:0/383774700 shutdown_connections 2024-03-25T19:24:55.707 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.704+0000 7f0fa0a33640 1 -- 172.21.15.6:0/383774700 >> 172.21.15.6:0/383774700 conn(0x7f0f9c0617a0 msgr2=0x7f0f9c061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:55.707 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.704+0000 7f0fa0a33640 1 -- 172.21.15.6:0/383774700 shutdown_connections 2024-03-25T19:24:55.707 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.704+0000 7f0fa0a33640 1 -- 172.21.15.6:0/383774700 wait complete. 2024-03-25T19:24:55.707 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.705+0000 7f0fa0a33640 10 monclient: build_initial_monmap 2024-03-25T19:24:55.707 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.705+0000 7f0fa0a33640 10 monclient: monmap: 2024-03-25T19:24:55.707 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:24:55.707 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:24:55.707 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:24:55.707 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:24:55.708 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:55.708 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:24:55.708 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:24:55.708 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:24:55.708 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:24:55.708 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:55.708 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.705+0000 7f0fa0a33640 1 librados: starting msgr at 2024-03-25T19:24:55.708 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.705+0000 7f0fa0a33640 1 librados: starting objecter 2024-03-25T19:24:55.708 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.705+0000 7f0fa0a33640 1 Processor -- start 2024-03-25T19:24:55.708 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.705+0000 7f0fa0a33640 1 -- start start 2024-03-25T19:24:55.708 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.705+0000 7f0fa0a33640 1 librados: setting wanted keys 2024-03-25T19:24:55.708 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.705+0000 7f0fa0a33640 1 librados: calling monclient init 2024-03-25T19:24:55.708 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.705+0000 7f0fa0a33640 10 monclient: init 2024-03-25T19:24:55.708 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.706+0000 7f0fa0a33640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:55.708 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.706+0000 7f0fa0a33640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:24:55.709 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.706+0000 7f0fa0a33640 10 monclient(hunting): picked mon.noname-b con 0x7f0f9c0b1b00 addr v1:172.21.15.192:6789/0 2024-03-25T19:24:55.709 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.706+0000 7f0fa0a33640 10 monclient(hunting): picked mon.noname-a con 0x7f0f9c0b44b0 addr v1:172.21.15.6:6789/0 2024-03-25T19:24:55.709 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.706+0000 7f0fa0a33640 10 monclient(hunting): picked mon.noname-c con 0x7f0f9c0b4e70 addr v1:172.21.15.6:6790/0 2024-03-25T19:24:55.709 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.706+0000 7f0fa0a33640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f9c1dbc80 con 0x7f0f9c0b44b0 2024-03-25T19:24:55.709 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.706+0000 7f0fa0a33640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f9c127ea0 con 0x7f0f9c0b1b00 2024-03-25T19:24:55.709 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.706+0000 7f0fa0a33640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f9c13c000 con 0x7f0f9c0b4e70 2024-03-25T19:24:55.709 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.706+0000 7f0fa0a33640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:55.709 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.706+0000 7f0fa0a33640 10 monclient(hunting): authenticate will time out at 6599.946195s 2024-03-25T19:24:55.717 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.717+0000 7f0f9a59c640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f0f9c0b1b00 0x7f0f9c1dae60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:43032/0 (socket says 172.21.15.6:43032) 2024-03-25T19:24:55.717 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.717+0000 7f0f9a59c640 1 -- 172.21.15.6:0/3242104149 learned_addr learned my addr 172.21.15.6:0/3242104149 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:24:55.719 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.719+0000 7f0f92ffd640 10 client.?.objecter ms_handle_connect 0x7f0f9c0b4e70 2024-03-25T19:24:55.719 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.719+0000 7f0f92ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:55.720 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.720+0000 7f0f92ffd640 10 client.?.objecter ms_handle_connect 0x7f0f9c0b1b00 2024-03-25T19:24:55.720 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.720+0000 7f0f92ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:55.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.724+0000 7f0f92ffd640 10 client.?.objecter ms_handle_connect 0x7f0f9c0b44b0 2024-03-25T19:24:55.723 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.724+0000 7f0f92ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:55.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.728+0000 7f0f92ffd640 1 -- 172.21.15.6:0/3242104149 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1582003529 0 0) 0x7f0f9c127ea0 con 0x7f0f9c0b1b00 2024-03-25T19:24:55.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.728+0000 7f0f92ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:24:55.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.728+0000 7f0f92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:55.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.728+0000 7f0f92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:55.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.728+0000 7f0f92ffd640 10 monclient(hunting): my global_id is 4723 2024-03-25T19:24:55.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.728+0000 7f0f92ffd640 1 -- 172.21.15.6:0/3242104149 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f7c004eb0 con 0x7f0f9c0b1b00 2024-03-25T19:24:55.735 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.736+0000 7f0f92ffd640 1 -- 172.21.15.6:0/3242104149 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4167109956 0 0) 0x7f0f9c1dbc80 con 0x7f0f9c0b44b0 2024-03-25T19:24:55.736 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.736+0000 7f0f92ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:24:55.736 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.736+0000 7f0f92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:55.736 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.736+0000 7f0f92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:55.736 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.736+0000 7f0f92ffd640 10 monclient(hunting): my global_id is 4713 2024-03-25T19:24:55.736 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.736+0000 7f0f92ffd640 1 -- 172.21.15.6:0/3242104149 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f9c127ea0 con 0x7f0f9c0b44b0 2024-03-25T19:24:55.742 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.742+0000 7f0f92ffd640 1 -- 172.21.15.6:0/3242104149 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1720069156 0 0) 0x7f0f7c004eb0 con 0x7f0f9c0b1b00 2024-03-25T19:24:55.742 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.742+0000 7f0f92ffd640 1 -- 172.21.15.6:0/3242104149 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f9c1dbc80 con 0x7f0f9c0b1b00 2024-03-25T19:24:55.743 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.742+0000 7f0f92ffd640 1 -- 172.21.15.6:0/3242104149 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f0f8c001970 con 0x7f0f9c0b1b00 2024-03-25T19:24:55.743 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.742+0000 7f0f92ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:55.743 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.742+0000 7f0f92ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:24:55.743 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.742+0000 7f0f92ffd640 10 monclient(hunting): dump: 2024-03-25T19:24:55.743 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:55.743 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:55.743 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:55.743 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:55.743 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:55.743 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:55.743 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:55.743 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:55.743 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:55.743 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:55.744 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.742+0000 7f0f92ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:24:55.751 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.751+0000 7f0f92ffd640 1 -- 172.21.15.6:0/3242104149 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 493879475 0 0) 0x7f0f9c13c000 con 0x7f0f9c0b4e70 2024-03-25T19:24:55.751 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.751+0000 7f0f92ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:24:55.751 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.751+0000 7f0f92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:55.751 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.751+0000 7f0f92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:55.752 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.751+0000 7f0f92ffd640 10 monclient(hunting): my global_id is 4712 2024-03-25T19:24:55.752 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.751+0000 7f0f92ffd640 1 -- 172.21.15.6:0/3242104149 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f7c004eb0 con 0x7f0f9c0b4e70 2024-03-25T19:24:55.752 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.751+0000 7f0f92ffd640 1 -- 172.21.15.6:0/3242104149 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2740653805 0 0) 0x7f0f9c127ea0 con 0x7f0f9c0b44b0 2024-03-25T19:24:55.752 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.751+0000 7f0f92ffd640 1 -- 172.21.15.6:0/3242104149 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f9c13c000 con 0x7f0f9c0b44b0 2024-03-25T19:24:55.752 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.751+0000 7f0f92ffd640 1 -- 172.21.15.6:0/3242104149 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f0f84001970 con 0x7f0f9c0b44b0 2024-03-25T19:24:55.752 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.751+0000 7f0f92ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:55.752 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.751+0000 7f0f92ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:24:55.752 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.751+0000 7f0f92ffd640 10 monclient(hunting): dump: 2024-03-25T19:24:55.752 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:55.752 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:55.752 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:55.752 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:55.752 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:55.752 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:55.753 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:55.753 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:55.753 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:55.753 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:55.762 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.761+0000 7f0f92ffd640 1 -- 172.21.15.6:0/3242104149 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 84254837 0 0) 0x7f0f9c1dbc80 con 0x7f0f9c0b1b00 2024-03-25T19:24:55.762 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.761+0000 7f0f92ffd640 1 -- 172.21.15.6:0/3242104149 >> v1:172.21.15.6:6790/0 conn(0x7f0f9c0b4e70 legacy=0x7f0f9c1db570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:55.762 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.761+0000 7f0f92ffd640 1 -- 172.21.15.6:0/3242104149 >> v1:172.21.15.6:6789/0 conn(0x7f0f9c0b44b0 legacy=0x7f0f9c1448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:55.762 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.761+0000 7f0f92ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:24:55.762 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.761+0000 7f0f92ffd640 1 monclient: found mon.b 2024-03-25T19:24:55.762 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.761+0000 7f0f92ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:55.762 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.761+0000 7f0f92ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:24:55.762 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.761+0000 7f0f92ffd640 1 -- 172.21.15.6:0/3242104149 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f9c13c2e0 con 0x7f0f9c0b1b00 2024-03-25T19:24:55.762 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.761+0000 7f0f92ffd640 10 monclient: _finish_auth 0 2024-03-25T19:24:55.762 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.761+0000 7f0f92ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:24:55.762 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.761+0000 7f0f92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:55.763 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.761+0000 7f0fa0a33640 5 monclient: authenticate success, global_id 4723 2024-03-25T19:24:55.763 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.761+0000 7f0fa0a33640 10 monclient: _renew_subs 2024-03-25T19:24:55.763 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.761+0000 7f0fa0a33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:24:55.763 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.761+0000 7f0fa0a33640 1 -- 172.21.15.6:0/3242104149 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f9c13c5c0 con 0x7f0f9c0b1b00 2024-03-25T19:24:55.763 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.761+0000 7f0fa0a33640 10 client.4723.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:24:55.763 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.761+0000 7f0fa0a33640 10 monclient: _renew_subs 2024-03-25T19:24:55.763 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.761+0000 7f0fa0a33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:24:55.763 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.761+0000 7f0fa0a33640 1 -- 172.21.15.6:0/3242104149 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f9c13cea0 con 0x7f0f9c0b1b00 2024-03-25T19:24:55.763 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.761+0000 7f0fa0a33640 1 librados: init done 2024-03-25T19:24:55.764 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.763+0000 7f0fa0a33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:55.764 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.763+0000 7f0fa0a33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:55.764 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.763+0000 7f0fa0a33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:24:55.764 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:55.763+0000 7f0fa0a33640 1 -- 172.21.15.6:0/3242104149 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f9c04f730 con 0x7f0f9c0b1b00 2024-03-25T19:24:56.043 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:56.043+0000 7f0f92ffd640 1 -- 172.21.15.6:0/3242104149 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f8c003610 con 0x7f0f9c0b1b00 2024-03-25T19:24:56.043 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:56.043+0000 7f0f92ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:56.043 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:56.043+0000 7f0f92ffd640 1 -- 172.21.15.6:0/3242104149 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f0f8c0039a0 con 0x7f0f9c0b1b00 2024-03-25T19:24:56.043 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:56.043+0000 7f0f92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:56.043 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:56.043+0000 7f0f92ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:24:56.043 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:56.043+0000 7f0f92ffd640 10 monclient: dump: 2024-03-25T19:24:56.043 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:24:56.043 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:24:56.043 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:56.043 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:24:56.043 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:56.044 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:24:56.044 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:24:56.044 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:24:56.044 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:24:56.044 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:24:56.044 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:56.043+0000 7f0f9959a640 4 set_mon_vals no callback set 2024-03-25T19:24:56.044 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:56.044+0000 7f0f92ffd640 1 -- 172.21.15.6:0/3242104149 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f0f8c003bc0 con 0x7f0f9c0b1b00 2024-03-25T19:24:56.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:56.044+0000 7f0f93fff640 10 monclient: get_auth_request con 0x7f0f7c06dfb0 auth_method 0 2024-03-25T19:24:56.045 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:56.045+0000 7f0f92ffd640 10 client.4723.objecter ms_handle_connect 0x7f0f7c06dfb0 2024-03-25T19:24:56.050 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:56.050+0000 7f0f92ffd640 1 -- 172.21.15.6:0/3242104149 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3970010106 0 0) 0x7f0f8c054780 con 0x7f0f9c0b1b00 2024-03-25T19:24:56.050 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:56.050+0000 7f0f92ffd640 10 client.4723.objecter ms_dispatch 0x7f0f9c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:24:56.050 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:56.050+0000 7f0f92ffd640 3 client.4723.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:24:56.050 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:56.050+0000 7f0f92ffd640 3 client.4723.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:24:56.050 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:56.050+0000 7f0f92ffd640 20 client.4723.objecter dump_active .. 0 homeless 2024-03-25T19:24:56.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:56.050+0000 7f0f92ffd640 1 -- 172.21.15.6:0/3242104149 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0f8c050290 con 0x7f0f9c0b1b00 2024-03-25T19:24:56.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:56.050+0000 7f0f92ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:56.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:56.050+0000 7f0f92ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:24:56.280 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:56.280+0000 7f0fa0a33640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:56.280 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:56.280+0000 7f0fa0a33640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:56.280 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:56.280+0000 7f0fa0a33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:24:56.280 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:56.280+0000 7f0fa0a33640 1 -- 172.21.15.6:0/3242104149 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f9c069f30 con 0x7f0f9c0b1b00 2024-03-25T19:24:56.432 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:56.432+0000 7f0f92ffd640 1 -- 172.21.15.6:0/3242104149 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3097771541) 0x7f0f8c0521c0 con 0x7f0f9c0b1b00 2024-03-25T19:24:56.432 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:56.432+0000 7f0f92ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:56.433 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:56.432+0000 7f0f92ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:24:56.433 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:24:56.433 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":11,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:24:09.307681+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:24:09.307681+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:24:56.435 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:56.435+0000 7f0fa0a33640 10 librados: watch_flush enter 2024-03-25T19:24:56.435 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:56.435+0000 7f0fa0a33640 10 librados: watch_flush exit 2024-03-25T19:24:56.435 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:56.435+0000 7f0fa0a33640 20 client.4723.objecter shutdown clearing up homeless session... 2024-03-25T19:24:56.435 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:56.435+0000 7f0fa0a33640 10 client.4723.objecter successfully canceled tick 2024-03-25T19:24:56.436 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:56.435+0000 7f0fa0a33640 1 -- 172.21.15.6:0/3242104149 >> v1:172.21.15.6:6800/2936652650 conn(0x7f0f7c06dfb0 legacy=0x7f0f7c070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:56.436 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:56.435+0000 7f0fa0a33640 10 monclient: shutdown 2024-03-25T19:24:56.436 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:56.435+0000 7f0fa0a33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:56.436 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:56.435+0000 7f0fa0a33640 1 -- 172.21.15.6:0/3242104149 >> v1:172.21.15.192:6789/0 conn(0x7f0f9c0b1b00 legacy=0x7f0f9c1dae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:56.436 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:56.435+0000 7f0fa0a33640 1 -- 172.21.15.6:0/3242104149 shutdown_connections 2024-03-25T19:24:56.436 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:56.435+0000 7f0fa0a33640 1 -- 172.21.15.6:0/3242104149 >> 172.21.15.6:0/3242104149 conn(0x7f0f9c0617a0 msgr2=0x7f0f9c0b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:56.436 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:56.435+0000 7f0fa0a33640 1 -- 172.21.15.6:0/3242104149 shutdown_connections 2024-03-25T19:24:56.436 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:56.435+0000 7f0fa0a33640 1 -- 172.21.15.6:0/3242104149 wait complete. 2024-03-25T19:24:56.436 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:24:56.435+0000 7f0fa0a33640 1 librados: shutdown 2024-03-25T19:24:56.449 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:25:03.255 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:25:03.376 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.376+0000 7f4504b7b640 10 monclient: get_monmap_and_config 2024-03-25T19:25:03.376 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.376+0000 7f4504b7b640 10 monclient: build_initial_monmap 2024-03-25T19:25:03.376 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.376+0000 7f4504b7b640 10 monclient: monmap: 2024-03-25T19:25:03.376 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:25:03.376 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:25:03.376 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:25:03.377210+0000 2024-03-25T19:25:03.376 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:25:03.377210+0000 2024-03-25T19:25:03.376 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:03.376 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:25:03.376 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:25:03.376 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:25:03.376 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:25:03.377 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:25:03.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.377+0000 7f4504b7b640 1 Processor -- start 2024-03-25T19:25:03.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.377+0000 7f4504b7b640 1 -- start start 2024-03-25T19:25:03.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.377+0000 7f4504b7b640 10 monclient: init 2024-03-25T19:25:03.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.377+0000 7f4504b7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:03.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.377+0000 7f4504b7b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:25:03.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.377+0000 7f4504b7b640 10 monclient(hunting): picked mon.noname-a con 0x7f45000b4e70 addr v1:172.21.15.6:6789/0 2024-03-25T19:25:03.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.377+0000 7f4504b7b640 10 monclient(hunting): picked mon.noname-b con 0x7f45000b44b0 addr v1:172.21.15.6:6790/0 2024-03-25T19:25:03.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.377+0000 7f4504b7b640 10 monclient(hunting): picked mon.noname-c con 0x7f45000b1b00 addr v1:172.21.15.192:6789/0 2024-03-25T19:25:03.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.378+0000 7f4504b7b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45000a96f0 con 0x7f45000b4e70 2024-03-25T19:25:03.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.378+0000 7f4504b7b640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45000ab0e0 con 0x7f45000b1b00 2024-03-25T19:25:03.379 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.378+0000 7f4504b7b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4500145f70 con 0x7f45000b44b0 2024-03-25T19:25:03.379 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.378+0000 7f4504b7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:03.379 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.378+0000 7f4504b7b640 10 monclient(hunting): authenticate will time out at 6607.617886s 2024-03-25T19:25:03.390 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.390+0000 7f44fed9d640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f45000b1b00 0x7f450014dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:41686/0 (socket says 172.21.15.6:41686) 2024-03-25T19:25:03.390 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.390+0000 7f44fed9d640 1 -- 172.21.15.6:0/2015340450 learned_addr learned my addr 172.21.15.6:0/2015340450 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:25:03.397 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.397+0000 7f44fdd9b640 1 -- 172.21.15.6:0/2015340450 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3336266668 0 0) 0x7f45000ab0e0 con 0x7f45000b1b00 2024-03-25T19:25:03.397 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.397+0000 7f44fdd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:25:03.397 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.397+0000 7f44fdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:03.397 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.397+0000 7f44fdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:03.397 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.397+0000 7f44fdd9b640 10 monclient(hunting): my global_id is 4726 2024-03-25T19:25:03.397 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.397+0000 7f44fdd9b640 1 -- 172.21.15.6:0/2015340450 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44ec003520 con 0x7f45000b1b00 2024-03-25T19:25:03.404 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.404+0000 7f44fdd9b640 1 -- 172.21.15.6:0/2015340450 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1439338562 0 0) 0x7f45000a96f0 con 0x7f45000b4e70 2024-03-25T19:25:03.404 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.404+0000 7f44fdd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:25:03.404 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.404+0000 7f44fdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:03.404 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.404+0000 7f44fdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:03.404 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.404+0000 7f44fdd9b640 10 monclient(hunting): my global_id is 4716 2024-03-25T19:25:03.404 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.404+0000 7f44fdd9b640 1 -- 172.21.15.6:0/2015340450 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45000ab0e0 con 0x7f45000b4e70 2024-03-25T19:25:03.415 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.415+0000 7f44fdd9b640 1 -- 172.21.15.6:0/2015340450 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3261559535 0 0) 0x7f45000ab0e0 con 0x7f45000b4e70 2024-03-25T19:25:03.415 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.415+0000 7f44fdd9b640 1 -- 172.21.15.6:0/2015340450 >> v1:172.21.15.6:6790/0 conn(0x7f45000b44b0 legacy=0x7f4500145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:03.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.415+0000 7f44fdd9b640 1 -- 172.21.15.6:0/2015340450 >> v1:172.21.15.192:6789/0 conn(0x7f45000b1b00 legacy=0x7f450014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:03.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.415+0000 7f44fdd9b640 10 monclient: _finish_hunting 0 2024-03-25T19:25:03.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.415+0000 7f44fdd9b640 1 monclient: found mon.noname-a 2024-03-25T19:25:03.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.415+0000 7f44fdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:03.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.415+0000 7f44fdd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-03-25T19:25:03.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.415+0000 7f44fdd9b640 1 -- 172.21.15.6:0/2015340450 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45000a9b80 con 0x7f45000b4e70 2024-03-25T19:25:03.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.415+0000 7f44fdd9b640 10 monclient: _finish_auth 0 2024-03-25T19:25:03.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.415+0000 7f44fdd9b640 10 monclient: _check_auth_tickets 2024-03-25T19:25:03.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.415+0000 7f44fdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:03.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.415+0000 7f44fdd9b640 1 -- 172.21.15.6:0/2015340450 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f44f0001670 con 0x7f45000b4e70 2024-03-25T19:25:03.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.415+0000 7f44fdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:03.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.415+0000 7f44fdd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:25:03.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.415+0000 7f44fdd9b640 10 monclient: dump: 2024-03-25T19:25:03.416 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:25:03.416 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:25:03.416 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:03.417 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:03.417 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:03.417 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:25:03.417 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:25:03.417 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:25:03.417 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:25:03.417 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:25:03.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.415+0000 7f4504b7b640 5 monclient: authenticate success, global_id 4716 2024-03-25T19:25:03.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.415+0000 7f4504b7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:03.420 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.420+0000 7f44fdd9b640 1 -- 172.21.15.6:0/2015340450 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44f00017d0 con 0x7f45000b4e70 2024-03-25T19:25:03.421 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.421+0000 7f44fdd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:03.421 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.421+0000 7f44fdd9b640 1 -- 172.21.15.6:0/2015340450 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f44f0001ba0 con 0x7f45000b4e70 2024-03-25T19:25:03.421 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.421+0000 7f44fdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:03.421 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.421+0000 7f44fdd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:25:03.421 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.421+0000 7f44fdd9b640 10 monclient: dump: 2024-03-25T19:25:03.421 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:25:03.421 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:25:03.421 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:03.421 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:03.421 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:03.421 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:25:03.421 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:25:03.422 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:25:03.422 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:25:03.422 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:25:03.422 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.421+0000 7f4504b7b640 10 monclient: get_monmap_and_config success 2024-03-25T19:25:03.422 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.421+0000 7f4504b7b640 4 set_mon_vals no callback set 2024-03-25T19:25:03.423 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.421+0000 7f4504b7b640 10 monclient: shutdown 2024-03-25T19:25:03.423 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.421+0000 7f4504b7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:03.423 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.421+0000 7f4504b7b640 1 -- 172.21.15.6:0/2015340450 >> v1:172.21.15.6:6789/0 conn(0x7f45000b4e70 legacy=0x7f450013f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:03.424 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.421+0000 7f4504b7b640 1 -- 172.21.15.6:0/2015340450 shutdown_connections 2024-03-25T19:25:03.424 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.421+0000 7f4504b7b640 1 -- 172.21.15.6:0/2015340450 >> 172.21.15.6:0/2015340450 conn(0x7f45000617a0 msgr2=0x7f4500061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:03.424 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.421+0000 7f4504b7b640 1 -- 172.21.15.6:0/2015340450 shutdown_connections 2024-03-25T19:25:03.424 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.422+0000 7f4504b7b640 1 -- 172.21.15.6:0/2015340450 wait complete. 2024-03-25T19:25:03.424 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.422+0000 7f4504b7b640 10 monclient: build_initial_monmap 2024-03-25T19:25:03.424 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.422+0000 7f4504b7b640 10 monclient: monmap: 2024-03-25T19:25:03.424 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:25:03.424 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:25:03.424 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:25:03.424 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:25:03.424 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:03.424 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:25:03.424 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:25:03.425 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:25:03.425 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:25:03.425 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:25:03.425 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.423+0000 7f4504b7b640 1 librados: starting msgr at 2024-03-25T19:25:03.425 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.423+0000 7f4504b7b640 1 librados: starting objecter 2024-03-25T19:25:03.425 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.423+0000 7f4504b7b640 1 Processor -- start 2024-03-25T19:25:03.425 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.423+0000 7f4504b7b640 1 -- start start 2024-03-25T19:25:03.425 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.423+0000 7f4504b7b640 1 librados: setting wanted keys 2024-03-25T19:25:03.425 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.423+0000 7f4504b7b640 1 librados: calling monclient init 2024-03-25T19:25:03.425 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.423+0000 7f4504b7b640 10 monclient: init 2024-03-25T19:25:03.426 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.424+0000 7f4504b7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:03.426 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.424+0000 7f4504b7b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:25:03.426 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.424+0000 7f4504b7b640 10 monclient(hunting): picked mon.noname-b con 0x7f45000b1b00 addr v1:172.21.15.192:6789/0 2024-03-25T19:25:03.427 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.424+0000 7f4504b7b640 10 monclient(hunting): picked mon.noname-a con 0x7f45000b44b0 addr v1:172.21.15.6:6789/0 2024-03-25T19:25:03.427 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.424+0000 7f4504b7b640 10 monclient(hunting): picked mon.noname-c con 0x7f45000b4e70 addr v1:172.21.15.6:6790/0 2024-03-25T19:25:03.427 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.424+0000 7f4504b7b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45001dbcf0 con 0x7f45000b44b0 2024-03-25T19:25:03.427 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.424+0000 7f4504b7b640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4500141260 con 0x7f45000b1b00 2024-03-25T19:25:03.427 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.424+0000 7f4504b7b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f450013cd80 con 0x7f45000b4e70 2024-03-25T19:25:03.427 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.424+0000 7f4504b7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:03.427 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.424+0000 7f4504b7b640 10 monclient(hunting): authenticate will time out at 6607.664622s 2024-03-25T19:25:03.434 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.434+0000 7f44fe59c640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f45000b1b00 0x7f450013a3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:41722/0 (socket says 172.21.15.6:41722) 2024-03-25T19:25:03.434 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.434+0000 7f44fe59c640 1 -- 172.21.15.6:0/3791803281 learned_addr learned my addr 172.21.15.6:0/3791803281 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:25:03.435 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.435+0000 7f44f6ffd640 10 client.?.objecter ms_handle_connect 0x7f45000b1b00 2024-03-25T19:25:03.435 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.435+0000 7f44f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:03.440 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.440+0000 7f44f6ffd640 1 -- 172.21.15.6:0/3791803281 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3620582855 0 0) 0x7f4500141260 con 0x7f45000b1b00 2024-03-25T19:25:03.440 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.440+0000 7f44f6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:25:03.440 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.440+0000 7f44f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:03.440 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.440+0000 7f44f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:03.440 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.440+0000 7f44f6ffd640 10 monclient(hunting): my global_id is 4729 2024-03-25T19:25:03.440 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.440+0000 7f44f6ffd640 1 -- 172.21.15.6:0/3791803281 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44e0004eb0 con 0x7f45000b1b00 2024-03-25T19:25:03.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.443+0000 7f44f6ffd640 10 client.?.objecter ms_handle_connect 0x7f45000b44b0 2024-03-25T19:25:03.443 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.443+0000 7f44f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:03.451 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.452+0000 7f44f6ffd640 10 client.?.objecter ms_handle_connect 0x7f45000b4e70 2024-03-25T19:25:03.452 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.452+0000 7f44f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:03.454 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.454+0000 7f44f6ffd640 1 -- 172.21.15.6:0/3791803281 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1104327646 0 0) 0x7f45001dbcf0 con 0x7f45000b44b0 2024-03-25T19:25:03.454 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.454+0000 7f44f6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:25:03.454 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.454+0000 7f44f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:03.454 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.454+0000 7f44f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:03.454 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.454+0000 7f44f6ffd640 10 monclient(hunting): my global_id is 4719 2024-03-25T19:25:03.454 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.454+0000 7f44f6ffd640 1 -- 172.21.15.6:0/3791803281 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4500141260 con 0x7f45000b44b0 2024-03-25T19:25:03.458 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.458+0000 7f44f6ffd640 1 -- 172.21.15.6:0/3791803281 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 331879307 0 0) 0x7f44e0004eb0 con 0x7f45000b1b00 2024-03-25T19:25:03.458 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.458+0000 7f44f6ffd640 1 -- 172.21.15.6:0/3791803281 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45001dbcf0 con 0x7f45000b1b00 2024-03-25T19:25:03.458 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.458+0000 7f44f6ffd640 1 -- 172.21.15.6:0/3791803281 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f44f0001630 con 0x7f45000b1b00 2024-03-25T19:25:03.458 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.458+0000 7f44f6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:03.458 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.458+0000 7f44f6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:25:03.458 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.458+0000 7f44f6ffd640 10 monclient(hunting): dump: 2024-03-25T19:25:03.459 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:25:03.459 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:25:03.459 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:03.459 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:03.459 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:03.459 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:25:03.459 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:25:03.459 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:25:03.459 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:25:03.459 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:25:03.459 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.458+0000 7f44f6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:25:03.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.474+0000 7f44f6ffd640 1 -- 172.21.15.6:0/3791803281 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3782894696 0 0) 0x7f45001dbcf0 con 0x7f45000b1b00 2024-03-25T19:25:03.474 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.474+0000 7f44f6ffd640 1 -- 172.21.15.6:0/3791803281 >> v1:172.21.15.6:6790/0 conn(0x7f45000b4e70 legacy=0x7f45001db5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:03.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.474+0000 7f44f6ffd640 1 -- 172.21.15.6:0/3791803281 >> v1:172.21.15.6:6789/0 conn(0x7f45000b44b0 legacy=0x7f45001daed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:03.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.474+0000 7f44f6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:25:03.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.474+0000 7f44f6ffd640 1 monclient: found mon.b 2024-03-25T19:25:03.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.474+0000 7f44f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:03.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.474+0000 7f44f6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:25:03.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.474+0000 7f44f6ffd640 1 -- 172.21.15.6:0/3791803281 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4500139000 con 0x7f45000b1b00 2024-03-25T19:25:03.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.474+0000 7f44f6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:25:03.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.474+0000 7f44f6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:25:03.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.474+0000 7f44f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:03.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.474+0000 7f4504b7b640 5 monclient: authenticate success, global_id 4729 2024-03-25T19:25:03.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.474+0000 7f4504b7b640 10 monclient: _renew_subs 2024-03-25T19:25:03.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.474+0000 7f4504b7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:25:03.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.474+0000 7f4504b7b640 1 -- 172.21.15.6:0/3791803281 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f450013b620 con 0x7f45000b1b00 2024-03-25T19:25:03.475 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.474+0000 7f4504b7b640 10 client.4729.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:25:03.476 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.474+0000 7f4504b7b640 10 monclient: _renew_subs 2024-03-25T19:25:03.476 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.474+0000 7f4504b7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:25:03.476 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.474+0000 7f4504b7b640 1 -- 172.21.15.6:0/3791803281 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f450013e970 con 0x7f45000b1b00 2024-03-25T19:25:03.476 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.474+0000 7f4504b7b640 1 librados: init done 2024-03-25T19:25:03.476 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.475+0000 7f4504b7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:03.476 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.475+0000 7f4504b7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:03.476 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.475+0000 7f4504b7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:25:03.477 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.475+0000 7f4504b7b640 1 -- 172.21.15.6:0/3791803281 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f450004f730 con 0x7f45000b1b00 2024-03-25T19:25:03.749 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.750+0000 7f44f6ffd640 1 -- 172.21.15.6:0/3791803281 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44f00038a0 con 0x7f45000b1b00 2024-03-25T19:25:03.750 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.750+0000 7f44f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:03.750 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.750+0000 7f44f6ffd640 1 -- 172.21.15.6:0/3791803281 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f44f0003c30 con 0x7f45000b1b00 2024-03-25T19:25:03.750 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.750+0000 7f44f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:03.750 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.750+0000 7f44f6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:25:03.750 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.750+0000 7f44f6ffd640 10 monclient: dump: 2024-03-25T19:25:03.750 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:25:03.750 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:25:03.750 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:03.750 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:03.750 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:03.750 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:25:03.751 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:25:03.751 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:25:03.751 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:25:03.751 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:25:03.751 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.750+0000 7f44fd59a640 4 set_mon_vals no callback set 2024-03-25T19:25:03.752 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.751+0000 7f44f6ffd640 1 -- 172.21.15.6:0/3791803281 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f44f0003e50 con 0x7f45000b1b00 2024-03-25T19:25:03.752 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.751+0000 7f44f7fff640 10 monclient: get_auth_request con 0x7f44e006dd10 auth_method 0 2024-03-25T19:25:03.752 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.752+0000 7f44f6ffd640 10 client.4729.objecter ms_handle_connect 0x7f44e006dd10 2024-03-25T19:25:03.756 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.757+0000 7f44f6ffd640 1 -- 172.21.15.6:0/3791803281 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3970010106 0 0) 0x7f44f0054870 con 0x7f45000b1b00 2024-03-25T19:25:03.757 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.757+0000 7f44f6ffd640 10 client.4729.objecter ms_dispatch 0x7f4500000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:25:03.757 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.757+0000 7f44f6ffd640 3 client.4729.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:25:03.757 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.757+0000 7f44f6ffd640 3 client.4729.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:25:03.757 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.757+0000 7f44f6ffd640 20 client.4729.objecter dump_active .. 0 homeless 2024-03-25T19:25:03.757 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.757+0000 7f44f6ffd640 1 -- 172.21.15.6:0/3791803281 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f44f0050380 con 0x7f45000b1b00 2024-03-25T19:25:03.757 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.757+0000 7f44f6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:03.757 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:03.757+0000 7f44f6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:25:04.025 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:04.025+0000 7f4504b7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:04.025 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:04.025+0000 7f4504b7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:04.025 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:04.025+0000 7f4504b7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:25:04.026 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:04.025+0000 7f4504b7b640 1 -- 172.21.15.6:0/3791803281 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4500139190 con 0x7f45000b1b00 2024-03-25T19:25:04.049 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:04.049+0000 7f44f6ffd640 1 -- 172.21.15.6:0/3791803281 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3097771541) 0x7f44f00522b0 con 0x7f45000b1b00 2024-03-25T19:25:04.049 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:04.049+0000 7f44f6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:04.049 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:04.049+0000 7f44f6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:25:04.049 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:25:04.049 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":11,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:24:09.307681+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:24:09.307681+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:25:04.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:04.051+0000 7f4504b7b640 10 librados: watch_flush enter 2024-03-25T19:25:04.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:04.051+0000 7f4504b7b640 10 librados: watch_flush exit 2024-03-25T19:25:04.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:04.051+0000 7f4504b7b640 20 client.4729.objecter shutdown clearing up homeless session... 2024-03-25T19:25:04.051 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:04.051+0000 7f4504b7b640 10 client.4729.objecter successfully canceled tick 2024-03-25T19:25:04.052 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:04.051+0000 7f4504b7b640 1 -- 172.21.15.6:0/3791803281 >> v1:172.21.15.6:6800/2936652650 conn(0x7f44e006dd10 legacy=0x7f44e00701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:04.052 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:04.051+0000 7f4504b7b640 10 monclient: shutdown 2024-03-25T19:25:04.052 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:04.051+0000 7f4504b7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:04.052 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:04.051+0000 7f4504b7b640 1 -- 172.21.15.6:0/3791803281 >> v1:172.21.15.192:6789/0 conn(0x7f45000b1b00 legacy=0x7f450013a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:04.052 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:04.051+0000 7f4504b7b640 1 -- 172.21.15.6:0/3791803281 shutdown_connections 2024-03-25T19:25:04.052 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:04.051+0000 7f4504b7b640 1 -- 172.21.15.6:0/3791803281 >> 172.21.15.6:0/3791803281 conn(0x7f45000617a0 msgr2=0x7f45000b6480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:04.052 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:04.051+0000 7f4504b7b640 1 -- 172.21.15.6:0/3791803281 shutdown_connections 2024-03-25T19:25:04.052 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:04.051+0000 7f4504b7b640 1 -- 172.21.15.6:0/3791803281 wait complete. 2024-03-25T19:25:04.052 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:04.051+0000 7f4504b7b640 1 librados: shutdown 2024-03-25T19:25:04.062 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:25:09.651 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:11:26.997 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:09.670 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:11:27.004 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:09.692 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:11:27.039 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:09.696 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:11:27.014 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:09.706 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:11:27.024 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:09.710 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:11:27.057 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:09.713 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:11:27.060 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:09.731 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:11:27.049 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:09.741 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:11:27.088 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:10.004 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:11:27.351 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:10.008 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:11:27.355 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:10.023 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:11:27.341 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:10.041 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:11:27.359 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:10.044 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:11:27.391 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:10.050 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:11:27.397 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:10.059 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:11:27.406 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:10.068 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:11:27.386 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:10.086 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:11:27.434 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:10.869 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:25:10.978 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:10.978+0000 7f9c7726e640 10 monclient: get_monmap_and_config 2024-03-25T19:25:10.979 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:10.978+0000 7f9c7726e640 10 monclient: build_initial_monmap 2024-03-25T19:25:10.979 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:10.979+0000 7f9c7726e640 10 monclient: monmap: 2024-03-25T19:25:10.979 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:25:10.979 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:25:10.979 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:25:10.979989+0000 2024-03-25T19:25:10.979 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:25:10.979989+0000 2024-03-25T19:25:10.979 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:10.979 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:25:10.979 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:25:10.979 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:25:10.979 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:25:10.980 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:25:10.981 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:10.979+0000 7f9c7726e640 1 Processor -- start 2024-03-25T19:25:10.981 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:10.979+0000 7f9c7726e640 1 -- start start 2024-03-25T19:25:10.981 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:10.979+0000 7f9c7726e640 10 monclient: init 2024-03-25T19:25:10.981 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:10.980+0000 7f9c7726e640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:10.981 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:10.980+0000 7f9c7726e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:25:10.981 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:10.980+0000 7f9c7726e640 10 monclient(hunting): picked mon.noname-c con 0x7f9c700b4e70 addr v1:172.21.15.192:6789/0 2024-03-25T19:25:10.981 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:10.980+0000 7f9c7726e640 10 monclient(hunting): picked mon.noname-a con 0x7f9c700b44b0 addr v1:172.21.15.6:6789/0 2024-03-25T19:25:10.982 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:10.980+0000 7f9c7726e640 10 monclient(hunting): picked mon.noname-b con 0x7f9c700b1b00 addr v1:172.21.15.6:6790/0 2024-03-25T19:25:10.982 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:10.980+0000 7f9c7726e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c700a96f0 con 0x7f9c700b44b0 2024-03-25T19:25:10.982 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:10.980+0000 7f9c7726e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c700ab0e0 con 0x7f9c700b4e70 2024-03-25T19:25:10.982 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:10.980+0000 7f9c7726e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c70145f70 con 0x7f9c700b1b00 2024-03-25T19:25:10.982 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:10.980+0000 7f9c7726e640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:10.982 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:10.980+0000 7f9c7726e640 10 monclient(hunting): authenticate will time out at 6615.220569s 2024-03-25T19:25:10.996 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:10.996+0000 7f9c74809640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9c700b44b0 0x7f9c70145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47290/0 (socket says 172.21.15.6:47290) 2024-03-25T19:25:10.996 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:10.996+0000 7f9c74809640 1 -- 172.21.15.6:0/821201711 learned_addr learned my addr 172.21.15.6:0/821201711 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:25:11.019 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.019+0000 7f9c67fff640 1 -- 172.21.15.6:0/821201711 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2220813276 0 0) 0x7f9c700a96f0 con 0x7f9c700b44b0 2024-03-25T19:25:11.019 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.019+0000 7f9c67fff640 10 monclient(hunting): _negotiate 2024-03-25T19:25:11.019 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.019+0000 7f9c67fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:11.019 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.019+0000 7f9c67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:11.019 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.019+0000 7f9c67fff640 10 monclient(hunting): my global_id is 4722 2024-03-25T19:25:11.019 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.019+0000 7f9c67fff640 1 -- 172.21.15.6:0/821201711 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c5c003520 con 0x7f9c700b44b0 2024-03-25T19:25:11.023 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.023+0000 7f9c67fff640 1 -- 172.21.15.6:0/821201711 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1808190082 0 0) 0x7f9c70145f70 con 0x7f9c700b1b00 2024-03-25T19:25:11.023 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.023+0000 7f9c67fff640 10 monclient(hunting): _negotiate 2024-03-25T19:25:11.023 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.023+0000 7f9c67fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:11.023 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.023+0000 7f9c67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:11.023 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.023+0000 7f9c67fff640 10 monclient(hunting): my global_id is 4721 2024-03-25T19:25:11.023 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.023+0000 7f9c67fff640 1 -- 172.21.15.6:0/821201711 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c700a96f0 con 0x7f9c700b1b00 2024-03-25T19:25:11.032 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.033+0000 7f9c67fff640 1 -- 172.21.15.6:0/821201711 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2484758603 0 0) 0x7f9c700ab0e0 con 0x7f9c700b4e70 2024-03-25T19:25:11.033 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.033+0000 7f9c67fff640 10 monclient(hunting): _negotiate 2024-03-25T19:25:11.033 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.033+0000 7f9c67fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:11.033 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.033+0000 7f9c67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:11.033 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.033+0000 7f9c67fff640 10 monclient(hunting): my global_id is 4732 2024-03-25T19:25:11.033 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.033+0000 7f9c67fff640 1 -- 172.21.15.6:0/821201711 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c70145f70 con 0x7f9c700b4e70 2024-03-25T19:25:11.042 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.041+0000 7f9c67fff640 1 -- 172.21.15.6:0/821201711 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1785718243 0 0) 0x7f9c700a96f0 con 0x7f9c700b1b00 2024-03-25T19:25:11.042 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.041+0000 7f9c67fff640 1 -- 172.21.15.6:0/821201711 >> v1:172.21.15.192:6789/0 conn(0x7f9c700b4e70 legacy=0x7f9c7013f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:11.042 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.041+0000 7f9c67fff640 1 -- 172.21.15.6:0/821201711 >> v1:172.21.15.6:6789/0 conn(0x7f9c700b44b0 legacy=0x7f9c70145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:11.042 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.041+0000 7f9c67fff640 10 monclient: _finish_hunting 0 2024-03-25T19:25:11.042 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.041+0000 7f9c67fff640 1 monclient: found mon.noname-b 2024-03-25T19:25:11.042 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.041+0000 7f9c67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:11.042 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.041+0000 7f9c67fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-03-25T19:25:11.042 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.041+0000 7f9c67fff640 1 -- 172.21.15.6:0/821201711 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c700a9b80 con 0x7f9c700b1b00 2024-03-25T19:25:11.042 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.041+0000 7f9c67fff640 10 monclient: _finish_auth 0 2024-03-25T19:25:11.042 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.041+0000 7f9c67fff640 10 monclient: _check_auth_tickets 2024-03-25T19:25:11.042 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.041+0000 7f9c67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:11.043 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.041+0000 7f9c67fff640 1 -- 172.21.15.6:0/821201711 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f9c680019f0 con 0x7f9c700b1b00 2024-03-25T19:25:11.043 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.041+0000 7f9c7726e640 5 monclient: authenticate success, global_id 4721 2024-03-25T19:25:11.043 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.041+0000 7f9c7726e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:11.043 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.041+0000 7f9c67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:11.043 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.041+0000 7f9c67fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:25:11.043 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.041+0000 7f9c67fff640 10 monclient: dump: 2024-03-25T19:25:11.043 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:25:11.043 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:25:11.043 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:11.043 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:11.043 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:11.043 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:25:11.043 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:25:11.043 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:25:11.043 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:25:11.044 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:25:11.044 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.041+0000 7f9c7726e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:11.053 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.054+0000 7f9c67fff640 1 -- 172.21.15.6:0/821201711 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c68001b50 con 0x7f9c700b1b00 2024-03-25T19:25:11.053 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.054+0000 7f9c67fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:11.054 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.054+0000 7f9c67fff640 1 -- 172.21.15.6:0/821201711 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f9c68001f60 con 0x7f9c700b1b00 2024-03-25T19:25:11.054 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.054+0000 7f9c67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:11.054 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.054+0000 7f9c67fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:25:11.054 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.054+0000 7f9c67fff640 10 monclient: dump: 2024-03-25T19:25:11.054 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:25:11.054 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:25:11.054 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:11.054 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:11.054 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:11.054 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:25:11.054 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:25:11.054 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:25:11.054 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:25:11.055 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:25:11.055 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.054+0000 7f9c7726e640 10 monclient: get_monmap_and_config success 2024-03-25T19:25:11.055 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.054+0000 7f9c7726e640 4 set_mon_vals no callback set 2024-03-25T19:25:11.056 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.054+0000 7f9c7726e640 10 monclient: shutdown 2024-03-25T19:25:11.056 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.054+0000 7f9c7726e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:11.056 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.054+0000 7f9c7726e640 1 -- 172.21.15.6:0/821201711 >> v1:172.21.15.6:6790/0 conn(0x7f9c700b1b00 legacy=0x7f9c7014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:11.056 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.055+0000 7f9c7726e640 1 -- 172.21.15.6:0/821201711 shutdown_connections 2024-03-25T19:25:11.057 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.055+0000 7f9c7726e640 1 -- 172.21.15.6:0/821201711 >> 172.21.15.6:0/821201711 conn(0x7f9c700617a0 msgr2=0x7f9c70061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:11.057 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.055+0000 7f9c7726e640 1 -- 172.21.15.6:0/821201711 shutdown_connections 2024-03-25T19:25:11.057 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.055+0000 7f9c7726e640 1 -- 172.21.15.6:0/821201711 wait complete. 2024-03-25T19:25:11.057 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.056+0000 7f9c7726e640 10 monclient: build_initial_monmap 2024-03-25T19:25:11.057 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.056+0000 7f9c7726e640 10 monclient: monmap: 2024-03-25T19:25:11.057 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:25:11.057 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:25:11.057 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:25:11.057 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:25:11.057 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:11.057 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:25:11.057 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:25:11.057 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:25:11.057 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:25:11.057 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:25:11.057 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.056+0000 7f9c7726e640 1 librados: starting msgr at 2024-03-25T19:25:11.058 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.056+0000 7f9c7726e640 1 librados: starting objecter 2024-03-25T19:25:11.058 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.056+0000 7f9c7726e640 1 Processor -- start 2024-03-25T19:25:11.058 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.056+0000 7f9c7726e640 1 -- start start 2024-03-25T19:25:11.058 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.056+0000 7f9c7726e640 1 librados: setting wanted keys 2024-03-25T19:25:11.058 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.056+0000 7f9c7726e640 1 librados: calling monclient init 2024-03-25T19:25:11.058 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.056+0000 7f9c7726e640 10 monclient: init 2024-03-25T19:25:11.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.057+0000 7f9c7726e640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:11.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.057+0000 7f9c7726e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:25:11.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.057+0000 7f9c7726e640 10 monclient(hunting): picked mon.noname-a con 0x7f9c700b1b00 addr v1:172.21.15.6:6789/0 2024-03-25T19:25:11.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.057+0000 7f9c7726e640 10 monclient(hunting): picked mon.noname-c con 0x7f9c700b44b0 addr v1:172.21.15.6:6790/0 2024-03-25T19:25:11.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.057+0000 7f9c7726e640 10 monclient(hunting): picked mon.noname-b con 0x7f9c700b4e70 addr v1:172.21.15.192:6789/0 2024-03-25T19:25:11.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.057+0000 7f9c7726e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c701d7a90 con 0x7f9c700b1b00 2024-03-25T19:25:11.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.057+0000 7f9c7726e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c70138010 con 0x7f9c700b4e70 2024-03-25T19:25:11.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.057+0000 7f9c7726e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c701d8630 con 0x7f9c700b44b0 2024-03-25T19:25:11.059 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.057+0000 7f9c7726e640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:11.060 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.057+0000 7f9c7726e640 10 monclient(hunting): authenticate will time out at 6615.297538s 2024-03-25T19:25:11.064 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.065+0000 7f9c7580b640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f9c700b4e70 0x7f9c701e22a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:42566/0 (socket says 172.21.15.6:42566) 2024-03-25T19:25:11.064 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.065+0000 7f9c7580b640 1 -- 172.21.15.6:0/1851307217 learned_addr learned my addr 172.21.15.6:0/1851307217 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:25:11.066 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.067+0000 7f9c65ffb640 10 client.?.objecter ms_handle_connect 0x7f9c700b4e70 2024-03-25T19:25:11.066 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.067+0000 7f9c65ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:11.068 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.069+0000 7f9c65ffb640 10 client.?.objecter ms_handle_connect 0x7f9c700b1b00 2024-03-25T19:25:11.068 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.069+0000 7f9c65ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:11.075 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.075+0000 7f9c65ffb640 10 client.?.objecter ms_handle_connect 0x7f9c700b44b0 2024-03-25T19:25:11.075 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.075+0000 7f9c65ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:11.081 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.082+0000 7f9c65ffb640 1 -- 172.21.15.6:0/1851307217 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2676749312 0 0) 0x7f9c701d7a90 con 0x7f9c700b1b00 2024-03-25T19:25:11.082 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.082+0000 7f9c65ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:25:11.082 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.082+0000 7f9c65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:11.082 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.082+0000 7f9c65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:11.082 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.082+0000 7f9c65ffb640 10 monclient(hunting): my global_id is 4725 2024-03-25T19:25:11.082 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.082+0000 7f9c65ffb640 1 -- 172.21.15.6:0/1851307217 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c54004eb0 con 0x7f9c700b1b00 2024-03-25T19:25:11.086 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.086+0000 7f9c65ffb640 1 -- 172.21.15.6:0/1851307217 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3336873998 0 0) 0x7f9c701d8630 con 0x7f9c700b44b0 2024-03-25T19:25:11.086 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.086+0000 7f9c65ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:25:11.086 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.086+0000 7f9c65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:11.086 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.086+0000 7f9c65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:11.087 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.086+0000 7f9c65ffb640 10 monclient(hunting): my global_id is 4724 2024-03-25T19:25:11.087 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.086+0000 7f9c65ffb640 1 -- 172.21.15.6:0/1851307217 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c701d7a90 con 0x7f9c700b44b0 2024-03-25T19:25:11.089 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.087+0000 7f9c65ffb640 1 -- 172.21.15.6:0/1851307217 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 674723792 0 0) 0x7f9c70138010 con 0x7f9c700b4e70 2024-03-25T19:25:11.089 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.087+0000 7f9c65ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:25:11.089 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.087+0000 7f9c65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:11.089 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.087+0000 7f9c65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:11.089 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.087+0000 7f9c65ffb640 10 monclient(hunting): my global_id is 4735 2024-03-25T19:25:11.090 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.087+0000 7f9c65ffb640 1 -- 172.21.15.6:0/1851307217 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c701d8630 con 0x7f9c700b4e70 2024-03-25T19:25:11.102 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.102+0000 7f9c65ffb640 1 -- 172.21.15.6:0/1851307217 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3764515768 0 0) 0x7f9c701d7a90 con 0x7f9c700b44b0 2024-03-25T19:25:11.102 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.102+0000 7f9c65ffb640 1 -- 172.21.15.6:0/1851307217 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c70138010 con 0x7f9c700b44b0 2024-03-25T19:25:11.102 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.102+0000 7f9c65ffb640 1 -- 172.21.15.6:0/1851307217 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f9c60001950 con 0x7f9c700b44b0 2024-03-25T19:25:11.102 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.102+0000 7f9c65ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:11.102 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.102+0000 7f9c65ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:25:11.103 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.102+0000 7f9c65ffb640 10 monclient(hunting): dump: 2024-03-25T19:25:11.103 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:25:11.103 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:25:11.103 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:11.103 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:11.103 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:11.103 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:25:11.103 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:25:11.103 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:25:11.103 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:25:11.103 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:25:11.103 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.102+0000 7f9c65ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:25:11.106 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.106+0000 7f9c65ffb640 1 -- 172.21.15.6:0/1851307217 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 641683320 0 0) 0x7f9c54004eb0 con 0x7f9c700b1b00 2024-03-25T19:25:11.106 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.106+0000 7f9c65ffb640 1 -- 172.21.15.6:0/1851307217 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c701d7a90 con 0x7f9c700b1b00 2024-03-25T19:25:11.106 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.106+0000 7f9c65ffb640 1 -- 172.21.15.6:0/1851307217 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f9c6c001850 con 0x7f9c700b1b00 2024-03-25T19:25:11.106 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.106+0000 7f9c65ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:11.106 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.106+0000 7f9c65ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:25:11.106 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.106+0000 7f9c65ffb640 10 monclient(hunting): dump: 2024-03-25T19:25:11.106 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:25:11.106 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:25:11.106 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:11.106 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:11.106 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:11.106 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:25:11.107 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:25:11.107 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:25:11.107 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:25:11.107 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:25:11.111 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.111+0000 7f9c65ffb640 1 -- 172.21.15.6:0/1851307217 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1163574913 0 0) 0x7f9c701d8630 con 0x7f9c700b4e70 2024-03-25T19:25:11.112 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.111+0000 7f9c65ffb640 1 -- 172.21.15.6:0/1851307217 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c54004eb0 con 0x7f9c700b4e70 2024-03-25T19:25:11.112 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.111+0000 7f9c65ffb640 1 -- 172.21.15.6:0/1851307217 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f9c68001ea0 con 0x7f9c700b4e70 2024-03-25T19:25:11.112 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.111+0000 7f9c65ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:11.112 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.111+0000 7f9c65ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:25:11.112 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.111+0000 7f9c65ffb640 10 monclient(hunting): dump: 2024-03-25T19:25:11.112 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:25:11.112 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:25:11.112 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:11.112 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:11.112 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:11.112 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:25:11.112 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:25:11.112 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:25:11.112 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:25:11.113 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:25:11.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.115+0000 7f9c65ffb640 1 -- 172.21.15.6:0/1851307217 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2692427919 0 0) 0x7f9c70138010 con 0x7f9c700b44b0 2024-03-25T19:25:11.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.115+0000 7f9c65ffb640 1 -- 172.21.15.6:0/1851307217 >> v1:172.21.15.192:6789/0 conn(0x7f9c700b4e70 legacy=0x7f9c701e22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:11.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.115+0000 7f9c65ffb640 1 -- 172.21.15.6:0/1851307217 >> v1:172.21.15.6:6789/0 conn(0x7f9c700b1b00 legacy=0x7f9c70135c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:11.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.115+0000 7f9c65ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:25:11.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.115+0000 7f9c65ffb640 1 monclient: found mon.c 2024-03-25T19:25:11.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.115+0000 7f9c65ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:11.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.115+0000 7f9c65ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:25:11.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.115+0000 7f9c65ffb640 1 -- 172.21.15.6:0/1851307217 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c701416f0 con 0x7f9c700b44b0 2024-03-25T19:25:11.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.115+0000 7f9c65ffb640 10 monclient: _finish_auth 0 2024-03-25T19:25:11.116 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.115+0000 7f9c65ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:25:11.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.115+0000 7f9c65ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:11.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.115+0000 7f9c7726e640 5 monclient: authenticate success, global_id 4724 2024-03-25T19:25:11.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.115+0000 7f9c7726e640 10 monclient: _renew_subs 2024-03-25T19:25:11.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.115+0000 7f9c7726e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:25:11.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.115+0000 7f9c7726e640 1 -- 172.21.15.6:0/1851307217 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c701419c0 con 0x7f9c700b44b0 2024-03-25T19:25:11.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.116+0000 7f9c7726e640 10 client.4724.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:25:11.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.116+0000 7f9c7726e640 10 monclient: _renew_subs 2024-03-25T19:25:11.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.116+0000 7f9c7726e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:25:11.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.116+0000 7f9c7726e640 1 -- 172.21.15.6:0/1851307217 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c701366c0 con 0x7f9c700b44b0 2024-03-25T19:25:11.117 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.116+0000 7f9c7726e640 1 librados: init done 2024-03-25T19:25:11.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.117+0000 7f9c7726e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:11.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.117+0000 7f9c7726e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:11.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.117+0000 7f9c7726e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:25:11.118 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.117+0000 7f9c7726e640 1 -- 172.21.15.6:0/1851307217 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c701d7860 con 0x7f9c700b44b0 2024-03-25T19:25:11.436 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.436+0000 7f9c65ffb640 1 -- 172.21.15.6:0/1851307217 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c60001d60 con 0x7f9c700b44b0 2024-03-25T19:25:11.437 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.436+0000 7f9c65ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:11.437 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.436+0000 7f9c65ffb640 1 -- 172.21.15.6:0/1851307217 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f9c600047e0 con 0x7f9c700b44b0 2024-03-25T19:25:11.437 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.436+0000 7f9c65ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:11.437 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.436+0000 7f9c65ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:25:11.437 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.436+0000 7f9c65ffb640 10 monclient: dump: 2024-03-25T19:25:11.437 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:25:11.437 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:25:11.437 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:11.437 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:11.437 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:11.437 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:25:11.437 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:25:11.437 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:25:11.437 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:25:11.438 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:25:11.438 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.436+0000 7f9c677fe640 4 set_mon_vals no callback set 2024-03-25T19:25:11.438 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.438+0000 7f9c65ffb640 1 -- 172.21.15.6:0/1851307217 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f9c6001a7b0 con 0x7f9c700b44b0 2024-03-25T19:25:11.438 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.438+0000 7f9c7500a640 10 monclient: get_auth_request con 0x7f9c5406e110 auth_method 0 2024-03-25T19:25:11.438 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.438+0000 7f9c65ffb640 10 client.4724.objecter ms_handle_connect 0x7f9c5406e110 2024-03-25T19:25:11.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.444+0000 7f9c65ffb640 1 -- 172.21.15.6:0/1851307217 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3970010106 0 0) 0x7f9c600545b0 con 0x7f9c700b44b0 2024-03-25T19:25:11.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.444+0000 7f9c65ffb640 10 client.4724.objecter ms_dispatch 0x7f9c70000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:25:11.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.444+0000 7f9c65ffb640 3 client.4724.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:25:11.444 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.444+0000 7f9c65ffb640 3 client.4724.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:25:11.445 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.444+0000 7f9c65ffb640 20 client.4724.objecter dump_active .. 0 homeless 2024-03-25T19:25:11.445 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.444+0000 7f9c65ffb640 1 -- 172.21.15.6:0/1851307217 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9c6004fef0 con 0x7f9c700b44b0 2024-03-25T19:25:11.445 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.444+0000 7f9c65ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:11.445 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.444+0000 7f9c65ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:25:11.698 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.698+0000 7f9c7726e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:11.698 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.698+0000 7f9c7726e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:11.698 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.698+0000 7f9c7726e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:25:11.699 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.698+0000 7f9c7726e640 1 -- 172.21.15.6:0/1851307217 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c701d7c40 con 0x7f9c700b44b0 2024-03-25T19:25:11.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.728+0000 7f9c65ffb640 1 -- 172.21.15.6:0/1851307217 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3097771541) 0x7f9c6001dbd0 con 0x7f9c700b44b0 2024-03-25T19:25:11.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.728+0000 7f9c65ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:11.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.728+0000 7f9c65ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:25:11.728 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:25:11.729 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":11,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:24:09.307681+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:24:09.307681+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:25:11.731 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.731+0000 7f9c7726e640 10 librados: watch_flush enter 2024-03-25T19:25:11.731 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.731+0000 7f9c7726e640 10 librados: watch_flush exit 2024-03-25T19:25:11.731 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.731+0000 7f9c7726e640 20 client.4724.objecter shutdown clearing up homeless session... 2024-03-25T19:25:11.731 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.731+0000 7f9c7726e640 10 client.4724.objecter successfully canceled tick 2024-03-25T19:25:11.732 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.731+0000 7f9c7726e640 1 -- 172.21.15.6:0/1851307217 >> v1:172.21.15.6:6800/2936652650 conn(0x7f9c5406e110 legacy=0x7f9c540705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:11.732 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.731+0000 7f9c7726e640 10 monclient: shutdown 2024-03-25T19:25:11.732 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.731+0000 7f9c7726e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:11.732 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.731+0000 7f9c7726e640 1 -- 172.21.15.6:0/1851307217 >> v1:172.21.15.6:6790/0 conn(0x7f9c700b44b0 legacy=0x7f9c701d7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:11.732 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.731+0000 7f9c7726e640 1 -- 172.21.15.6:0/1851307217 shutdown_connections 2024-03-25T19:25:11.732 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.731+0000 7f9c7726e640 1 -- 172.21.15.6:0/1851307217 >> 172.21.15.6:0/1851307217 conn(0x7f9c700617a0 msgr2=0x7f9c700b59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:11.732 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.731+0000 7f9c7726e640 1 -- 172.21.15.6:0/1851307217 shutdown_connections 2024-03-25T19:25:11.732 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.731+0000 7f9c7726e640 1 -- 172.21.15.6:0/1851307217 wait complete. 2024-03-25T19:25:11.732 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:11.731+0000 7f9c7726e640 1 librados: shutdown 2024-03-25T19:25:11.746 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:25:18.550 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:25:18.668 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.669+0000 7f72174fb640 10 monclient: get_monmap_and_config 2024-03-25T19:25:18.669 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.669+0000 7f72174fb640 10 monclient: build_initial_monmap 2024-03-25T19:25:18.669 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.669+0000 7f72174fb640 10 monclient: monmap: 2024-03-25T19:25:18.669 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:25:18.669 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:25:18.669 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:25:18.670251+0000 2024-03-25T19:25:18.669 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:25:18.670251+0000 2024-03-25T19:25:18.669 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:18.669 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:25:18.669 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:25:18.669 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:25:18.669 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:25:18.669 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:25:18.670 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.670+0000 7f72174fb640 1 Processor -- start 2024-03-25T19:25:18.670 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.670+0000 7f72174fb640 1 -- start start 2024-03-25T19:25:18.670 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.670+0000 7f72174fb640 10 monclient: init 2024-03-25T19:25:18.671 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.671+0000 7f72174fb640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:18.671 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.671+0000 7f72174fb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:25:18.671 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.671+0000 7f72174fb640 10 monclient(hunting): picked mon.noname-c con 0x7f72100b4e70 addr v1:172.21.15.192:6789/0 2024-03-25T19:25:18.671 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.671+0000 7f72174fb640 10 monclient(hunting): picked mon.noname-b con 0x7f72100b44b0 addr v1:172.21.15.6:6790/0 2024-03-25T19:25:18.671 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.671+0000 7f72174fb640 10 monclient(hunting): picked mon.noname-a con 0x7f72100b1b00 addr v1:172.21.15.6:6789/0 2024-03-25T19:25:18.671 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.671+0000 7f72174fb640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72100a96f0 con 0x7f72100b1b00 2024-03-25T19:25:18.671 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.671+0000 7f72174fb640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72100ab0e0 con 0x7f72100b4e70 2024-03-25T19:25:18.671 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.671+0000 7f72174fb640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7210145f70 con 0x7f72100b44b0 2024-03-25T19:25:18.671 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.671+0000 7f72174fb640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:18.671 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.671+0000 7f72174fb640 10 monclient(hunting): authenticate will time out at 6622.910967s 2024-03-25T19:25:18.680 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.681+0000 7f7215a98640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f72100b1b00 0x7f721014dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35148/0 (socket says 172.21.15.6:35148) 2024-03-25T19:25:18.681 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.681+0000 7f7215a98640 1 -- 172.21.15.6:0/283386856 learned_addr learned my addr 172.21.15.6:0/283386856 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:25:18.694 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.694+0000 7f7207fff640 1 -- 172.21.15.6:0/283386856 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 735036271 0 0) 0x7f72100a96f0 con 0x7f72100b1b00 2024-03-25T19:25:18.694 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.694+0000 7f7207fff640 10 monclient(hunting): _negotiate 2024-03-25T19:25:18.694 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.694+0000 7f7207fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:18.694 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.694+0000 7f7207fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:18.694 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.694+0000 7f7207fff640 10 monclient(hunting): my global_id is 4728 2024-03-25T19:25:18.694 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.694+0000 7f7207fff640 1 -- 172.21.15.6:0/283386856 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71fc003520 con 0x7f72100b1b00 2024-03-25T19:25:18.701 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.701+0000 7f7207fff640 1 -- 172.21.15.6:0/283386856 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2167107028 0 0) 0x7f72100ab0e0 con 0x7f72100b4e70 2024-03-25T19:25:18.701 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.701+0000 7f7207fff640 10 monclient(hunting): _negotiate 2024-03-25T19:25:18.701 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.701+0000 7f7207fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:18.701 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.701+0000 7f7207fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:18.701 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.701+0000 7f7207fff640 10 monclient(hunting): my global_id is 4738 2024-03-25T19:25:18.701 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.701+0000 7f7207fff640 1 -- 172.21.15.6:0/283386856 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72100a96f0 con 0x7f72100b4e70 2024-03-25T19:25:18.707 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.707+0000 7f7207fff640 1 -- 172.21.15.6:0/283386856 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2536903307 0 0) 0x7f7210145f70 con 0x7f72100b44b0 2024-03-25T19:25:18.707 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.707+0000 7f7207fff640 10 monclient(hunting): _negotiate 2024-03-25T19:25:18.707 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.707+0000 7f7207fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:18.707 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.707+0000 7f7207fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:18.707 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.707+0000 7f7207fff640 10 monclient(hunting): my global_id is 4727 2024-03-25T19:25:18.707 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.707+0000 7f7207fff640 1 -- 172.21.15.6:0/283386856 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72100ab0e0 con 0x7f72100b44b0 2024-03-25T19:25:18.715 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.715+0000 7f7207fff640 1 -- 172.21.15.6:0/283386856 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2897870098 0 0) 0x7f72100a96f0 con 0x7f72100b4e70 2024-03-25T19:25:18.715 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.715+0000 7f7207fff640 1 -- 172.21.15.6:0/283386856 >> v1:172.21.15.6:6790/0 conn(0x7f72100b44b0 legacy=0x7f7210145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:18.715 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.715+0000 7f7207fff640 1 -- 172.21.15.6:0/283386856 >> v1:172.21.15.6:6789/0 conn(0x7f72100b1b00 legacy=0x7f721014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:18.715 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.715+0000 7f7207fff640 10 monclient: _finish_hunting 0 2024-03-25T19:25:18.716 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.715+0000 7f7207fff640 1 monclient: found mon.noname-c 2024-03-25T19:25:18.716 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.715+0000 7f7207fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:18.716 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.715+0000 7f7207fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-03-25T19:25:18.716 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.715+0000 7f7207fff640 1 -- 172.21.15.6:0/283386856 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72100a9b80 con 0x7f72100b4e70 2024-03-25T19:25:18.716 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.715+0000 7f7207fff640 10 monclient: _finish_auth 0 2024-03-25T19:25:18.716 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.715+0000 7f7207fff640 10 monclient: _check_auth_tickets 2024-03-25T19:25:18.716 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.715+0000 7f7207fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:18.716 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.715+0000 7f7207fff640 1 -- 172.21.15.6:0/283386856 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f720c001670 con 0x7f72100b4e70 2024-03-25T19:25:18.716 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.715+0000 7f72174fb640 5 monclient: authenticate success, global_id 4738 2024-03-25T19:25:18.716 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.715+0000 7f72174fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:18.716 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.715+0000 7f7207fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:18.716 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.715+0000 7f7207fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:25:18.716 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.715+0000 7f7207fff640 10 monclient: dump: 2024-03-25T19:25:18.716 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:25:18.716 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:25:18.717 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:18.717 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:18.717 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:18.717 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:25:18.717 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:25:18.717 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:25:18.717 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:25:18.717 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:25:18.717 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.715+0000 7f72174fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:18.726 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.726+0000 7f7207fff640 1 -- 172.21.15.6:0/283386856 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f720c0017d0 con 0x7f72100b4e70 2024-03-25T19:25:18.726 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.726+0000 7f7207fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:18.726 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.726+0000 7f7207fff640 1 -- 172.21.15.6:0/283386856 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f720c001be0 con 0x7f72100b4e70 2024-03-25T19:25:18.726 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.726+0000 7f7207fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:18.726 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.726+0000 7f7207fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:25:18.726 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.726+0000 7f7207fff640 10 monclient: dump: 2024-03-25T19:25:18.726 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:25:18.726 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:25:18.726 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:18.727 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:18.727 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:18.727 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:25:18.727 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:25:18.727 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:25:18.727 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:25:18.727 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:25:18.727 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.726+0000 7f72174fb640 10 monclient: get_monmap_and_config success 2024-03-25T19:25:18.727 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.726+0000 7f72174fb640 4 set_mon_vals no callback set 2024-03-25T19:25:18.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.727+0000 7f72174fb640 10 monclient: shutdown 2024-03-25T19:25:18.729 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.727+0000 7f72174fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:18.729 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.727+0000 7f72174fb640 1 -- 172.21.15.6:0/283386856 >> v1:172.21.15.192:6789/0 conn(0x7f72100b4e70 legacy=0x7f721013f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:18.729 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.727+0000 7f72174fb640 1 -- 172.21.15.6:0/283386856 shutdown_connections 2024-03-25T19:25:18.729 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.727+0000 7f72174fb640 1 -- 172.21.15.6:0/283386856 >> 172.21.15.6:0/283386856 conn(0x7f72100617a0 msgr2=0x7f7210061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:18.729 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.727+0000 7f72174fb640 1 -- 172.21.15.6:0/283386856 shutdown_connections 2024-03-25T19:25:18.729 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.727+0000 7f72174fb640 1 -- 172.21.15.6:0/283386856 wait complete. 2024-03-25T19:25:18.729 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.728+0000 7f72174fb640 10 monclient: build_initial_monmap 2024-03-25T19:25:18.729 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.728+0000 7f72174fb640 10 monclient: monmap: 2024-03-25T19:25:18.729 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:25:18.729 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:25:18.729 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:25:18.729 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:25:18.729 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:18.729 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:25:18.729 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:25:18.730 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:25:18.730 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:25:18.730 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:25:18.730 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.728+0000 7f72174fb640 1 librados: starting msgr at 2024-03-25T19:25:18.730 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.728+0000 7f72174fb640 1 librados: starting objecter 2024-03-25T19:25:18.730 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.729+0000 7f72174fb640 1 Processor -- start 2024-03-25T19:25:18.730 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.729+0000 7f72174fb640 1 -- start start 2024-03-25T19:25:18.730 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.729+0000 7f72174fb640 1 librados: setting wanted keys 2024-03-25T19:25:18.730 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.729+0000 7f72174fb640 1 librados: calling monclient init 2024-03-25T19:25:18.730 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.729+0000 7f72174fb640 10 monclient: init 2024-03-25T19:25:18.731 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.729+0000 7f72174fb640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:18.731 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.729+0000 7f72174fb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:25:18.731 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.729+0000 7f72174fb640 10 monclient(hunting): picked mon.noname-b con 0x7f72100b1b00 addr v1:172.21.15.192:6789/0 2024-03-25T19:25:18.731 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.730+0000 7f72174fb640 10 monclient(hunting): picked mon.noname-a con 0x7f72100b44b0 addr v1:172.21.15.6:6789/0 2024-03-25T19:25:18.731 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.730+0000 7f72174fb640 10 monclient(hunting): picked mon.noname-c con 0x7f72100b4e70 addr v1:172.21.15.6:6790/0 2024-03-25T19:25:18.731 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.730+0000 7f72174fb640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7210136bd0 con 0x7f72100b44b0 2024-03-25T19:25:18.731 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.730+0000 7f72174fb640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7210137ef0 con 0x7f72100b1b00 2024-03-25T19:25:18.732 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.730+0000 7f72174fb640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72101db0e0 con 0x7f72100b4e70 2024-03-25T19:25:18.732 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.730+0000 7f72174fb640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:18.732 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.730+0000 7f72174fb640 10 monclient(hunting): authenticate will time out at 6622.969983s 2024-03-25T19:25:18.736 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.736+0000 7f7215a98640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f72100b4e70 0x7f72101d4bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:47440/0 (socket says 172.21.15.6:47440) 2024-03-25T19:25:18.736 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.736+0000 7f7215a98640 1 -- 172.21.15.6:0/3985065820 learned_addr learned my addr 172.21.15.6:0/3985065820 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:25:18.736 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.736+0000 7f7205ffb640 10 client.?.objecter ms_handle_connect 0x7f72100b4e70 2024-03-25T19:25:18.736 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.736+0000 7f7205ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:18.742 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.742+0000 7f7205ffb640 10 client.?.objecter ms_handle_connect 0x7f72100b1b00 2024-03-25T19:25:18.742 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.742+0000 7f7205ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:18.743 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.743+0000 7f7205ffb640 1 -- 172.21.15.6:0/3985065820 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1471982117 0 0) 0x7f72101db0e0 con 0x7f72100b4e70 2024-03-25T19:25:18.743 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.743+0000 7f7205ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:25:18.743 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.743+0000 7f7205ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:18.743 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.743+0000 7f7205ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:18.744 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.743+0000 7f7205ffb640 10 monclient(hunting): my global_id is 4730 2024-03-25T19:25:18.744 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.743+0000 7f7205ffb640 1 -- 172.21.15.6:0/3985065820 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71f4004eb0 con 0x7f72100b4e70 2024-03-25T19:25:18.747 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.748+0000 7f7205ffb640 10 client.?.objecter ms_handle_connect 0x7f72100b44b0 2024-03-25T19:25:18.748 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.748+0000 7f7205ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:18.758 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.758+0000 7f7205ffb640 1 -- 172.21.15.6:0/3985065820 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2351865238 0 0) 0x7f71f4004eb0 con 0x7f72100b4e70 2024-03-25T19:25:18.758 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.758+0000 7f7205ffb640 1 -- 172.21.15.6:0/3985065820 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72101db0e0 con 0x7f72100b4e70 2024-03-25T19:25:18.758 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.758+0000 7f7205ffb640 1 -- 172.21.15.6:0/3985065820 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f7208001cd0 con 0x7f72100b4e70 2024-03-25T19:25:18.758 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.758+0000 7f7205ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:18.758 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.758+0000 7f7205ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:25:18.758 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.758+0000 7f7205ffb640 10 monclient(hunting): dump: 2024-03-25T19:25:18.758 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:25:18.758 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:25:18.758 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:18.759 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:18.759 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:18.759 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:25:18.759 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:25:18.759 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:25:18.759 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:25:18.759 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:25:18.759 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.758+0000 7f7205ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:25:18.765 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.766+0000 7f7205ffb640 1 -- 172.21.15.6:0/3985065820 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 594316526 0 0) 0x7f7210137ef0 con 0x7f72100b1b00 2024-03-25T19:25:18.766 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.766+0000 7f7205ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:25:18.766 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.766+0000 7f7205ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:18.766 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.766+0000 7f7205ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:18.766 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.766+0000 7f7205ffb640 10 monclient(hunting): my global_id is 4741 2024-03-25T19:25:18.766 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.766+0000 7f7205ffb640 1 -- 172.21.15.6:0/3985065820 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71f4004eb0 con 0x7f72100b1b00 2024-03-25T19:25:18.770 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.770+0000 7f7205ffb640 1 -- 172.21.15.6:0/3985065820 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 117286808 0 0) 0x7f7210136bd0 con 0x7f72100b44b0 2024-03-25T19:25:18.770 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.770+0000 7f7205ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:25:18.770 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.770+0000 7f7205ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:18.771 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.770+0000 7f7205ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:18.771 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.770+0000 7f7205ffb640 10 monclient(hunting): my global_id is 4731 2024-03-25T19:25:18.771 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.771+0000 7f7205ffb640 1 -- 172.21.15.6:0/3985065820 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7210137ef0 con 0x7f72100b44b0 2024-03-25T19:25:18.772 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.771+0000 7f7205ffb640 1 -- 172.21.15.6:0/3985065820 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 564835561 0 0) 0x7f72101db0e0 con 0x7f72100b4e70 2024-03-25T19:25:18.772 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.771+0000 7f7205ffb640 1 -- 172.21.15.6:0/3985065820 >> v1:172.21.15.192:6789/0 conn(0x7f72100b1b00 legacy=0x7f72101364c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:18.772 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.771+0000 7f7205ffb640 1 -- 172.21.15.6:0/3985065820 >> v1:172.21.15.6:6789/0 conn(0x7f72100b44b0 legacy=0x7f72101db600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:18.772 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.771+0000 7f7205ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:25:18.772 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.771+0000 7f7205ffb640 1 monclient: found mon.c 2024-03-25T19:25:18.772 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.771+0000 7f7205ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:18.772 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.771+0000 7f7205ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:25:18.772 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.771+0000 7f7205ffb640 1 -- 172.21.15.6:0/3985065820 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72101dbd10 con 0x7f72100b4e70 2024-03-25T19:25:18.772 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.771+0000 7f7205ffb640 10 monclient: _finish_auth 0 2024-03-25T19:25:18.772 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.771+0000 7f7205ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:25:18.772 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.771+0000 7f7205ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:18.772 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.771+0000 7f72174fb640 5 monclient: authenticate success, global_id 4730 2024-03-25T19:25:18.773 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.771+0000 7f72174fb640 10 monclient: _renew_subs 2024-03-25T19:25:18.773 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.771+0000 7f72174fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:25:18.773 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.771+0000 7f72174fb640 1 -- 172.21.15.6:0/3985065820 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72101d4540 con 0x7f72100b4e70 2024-03-25T19:25:18.773 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.771+0000 7f72174fb640 10 client.4730.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:25:18.773 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.771+0000 7f72174fb640 10 monclient: _renew_subs 2024-03-25T19:25:18.773 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.771+0000 7f72174fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:25:18.773 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.771+0000 7f72174fb640 1 -- 172.21.15.6:0/3985065820 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7210137020 con 0x7f72100b4e70 2024-03-25T19:25:18.773 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.771+0000 7f72174fb640 1 librados: init done 2024-03-25T19:25:18.773 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.773+0000 7f72174fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:18.773 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.773+0000 7f72174fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:18.774 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.773+0000 7f72174fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:25:18.774 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:18.773+0000 7f72174fb640 1 -- 172.21.15.6:0/3985065820 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72101daf40 con 0x7f72100b4e70 2024-03-25T19:25:19.093 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:19.093+0000 7f7205ffb640 1 -- 172.21.15.6:0/3985065820 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f72080020e0 con 0x7f72100b4e70 2024-03-25T19:25:19.093 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:19.093+0000 7f7205ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:19.094 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:19.093+0000 7f7205ffb640 1 -- 172.21.15.6:0/3985065820 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7f7208004be0 con 0x7f72100b4e70 2024-03-25T19:25:19.094 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:19.093+0000 7f7205ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:19.094 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:19.093+0000 7f7205ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:25:19.094 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:19.093+0000 7f7205ffb640 10 monclient: dump: 2024-03-25T19:25:19.094 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:25:19.094 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:25:19.094 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:19.094 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:19.094 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:19.094 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:25:19.094 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:25:19.094 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:25:19.094 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:25:19.094 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:25:19.094 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:19.093+0000 7f72077fe640 4 set_mon_vals no callback set 2024-03-25T19:25:19.095 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:19.094+0000 7f7205ffb640 1 -- 172.21.15.6:0/3985065820 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7f7208004dc0 con 0x7f72100b4e70 2024-03-25T19:25:19.095 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:19.095+0000 7f7215297640 10 monclient: get_auth_request con 0x7f71f406dea0 auth_method 0 2024-03-25T19:25:19.095 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:19.095+0000 7f7205ffb640 10 client.4730.objecter ms_handle_connect 0x7f71f406dea0 2024-03-25T19:25:19.101 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:19.101+0000 7f7205ffb640 1 -- 172.21.15.6:0/3985065820 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3970010106 0 0) 0x7f72080549a0 con 0x7f72100b4e70 2024-03-25T19:25:19.101 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:19.101+0000 7f7205ffb640 10 client.4730.objecter ms_dispatch 0x7f7210000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:25:19.101 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:19.101+0000 7f7205ffb640 3 client.4730.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:25:19.101 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:19.101+0000 7f7205ffb640 3 client.4730.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:25:19.102 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:19.101+0000 7f7205ffb640 20 client.4730.objecter dump_active .. 0 homeless 2024-03-25T19:25:19.102 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:19.101+0000 7f7205ffb640 1 -- 172.21.15.6:0/3985065820 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f72080502e0 con 0x7f72100b4e70 2024-03-25T19:25:19.102 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:19.101+0000 7f7205ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:19.102 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:19.101+0000 7f7205ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:25:19.351 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:19.351+0000 7f72174fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:19.352 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:19.351+0000 7f72174fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:19.352 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:19.351+0000 7f72174fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-03-25T19:25:19.352 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:19.351+0000 7f72174fb640 1 -- 172.21.15.6:0/3985065820 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72101dbea0 con 0x7f72100b4e70 2024-03-25T19:25:19.374 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:19.374+0000 7f7205ffb640 1 -- 172.21.15.6:0/3985065820 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3097771541) 0x7f72101dbea0 con 0x7f72100b4e70 2024-03-25T19:25:19.374 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:19.374+0000 7f7205ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:19.374 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:19.374+0000 7f7205ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:25:19.374 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:25:19.374 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":11,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:24:09.307681+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:24:09.307681+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:25:19.377 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:19.376+0000 7f72174fb640 10 librados: watch_flush enter 2024-03-25T19:25:19.377 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:19.376+0000 7f72174fb640 10 librados: watch_flush exit 2024-03-25T19:25:19.377 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:19.376+0000 7f72174fb640 20 client.4730.objecter shutdown clearing up homeless session... 2024-03-25T19:25:19.377 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:19.376+0000 7f72174fb640 10 client.4730.objecter successfully canceled tick 2024-03-25T19:25:19.377 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:19.376+0000 7f72174fb640 1 -- 172.21.15.6:0/3985065820 >> v1:172.21.15.6:6800/2936652650 conn(0x7f71f406dea0 legacy=0x7f71f4070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:19.377 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:19.376+0000 7f72174fb640 10 monclient: shutdown 2024-03-25T19:25:19.377 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:19.376+0000 7f72174fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:19.377 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:19.376+0000 7f72174fb640 1 -- 172.21.15.6:0/3985065820 >> v1:172.21.15.6:6790/0 conn(0x7f72100b4e70 legacy=0x7f72101d4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:19.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:19.377+0000 7f72174fb640 1 -- 172.21.15.6:0/3985065820 shutdown_connections 2024-03-25T19:25:19.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:19.377+0000 7f72174fb640 1 -- 172.21.15.6:0/3985065820 >> 172.21.15.6:0/3985065820 conn(0x7f72100617a0 msgr2=0x7f72100b6620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:19.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:19.377+0000 7f72174fb640 1 -- 172.21.15.6:0/3985065820 shutdown_connections 2024-03-25T19:25:19.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:19.377+0000 7f72174fb640 1 -- 172.21.15.6:0/3985065820 wait complete. 2024-03-25T19:25:19.378 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:19.377+0000 7f72174fb640 1 librados: shutdown 2024-03-25T19:25:19.388 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:25:26.195 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:25:26.311 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.311+0000 7fb9ec3a3640 10 monclient: get_monmap_and_config 2024-03-25T19:25:26.311 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.311+0000 7fb9ec3a3640 10 monclient: build_initial_monmap 2024-03-25T19:25:26.311 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.311+0000 7fb9ec3a3640 10 monclient: monmap: 2024-03-25T19:25:26.311 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:25:26.311 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:25:26.311 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:25:26.312288+0000 2024-03-25T19:25:26.311 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:25:26.312288+0000 2024-03-25T19:25:26.312 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:26.312 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:25:26.312 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:25:26.312 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:25:26.312 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:25:26.312 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:25:26.312 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.312+0000 7fb9ec3a3640 1 Processor -- start 2024-03-25T19:25:26.312 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.312+0000 7fb9ec3a3640 1 -- start start 2024-03-25T19:25:26.312 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.312+0000 7fb9ec3a3640 10 monclient: init 2024-03-25T19:25:26.313 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.313+0000 7fb9ec3a3640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:26.313 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.313+0000 7fb9ec3a3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:25:26.313 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.313+0000 7fb9ec3a3640 10 monclient(hunting): picked mon.noname-a con 0x7fb9e40b1ed0 addr v1:172.21.15.6:6789/0 2024-03-25T19:25:26.313 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.313+0000 7fb9ec3a3640 10 monclient(hunting): picked mon.noname-c con 0x7fb9e4071db0 addr v1:172.21.15.192:6789/0 2024-03-25T19:25:26.314 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.313+0000 7fb9ec3a3640 10 monclient(hunting): picked mon.noname-b con 0x7fb9e4143f90 addr v1:172.21.15.6:6790/0 2024-03-25T19:25:26.314 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.313+0000 7fb9ec3a3640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e406eef0 con 0x7fb9e40b1ed0 2024-03-25T19:25:26.314 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.313+0000 7fb9ec3a3640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e406f1d0 con 0x7fb9e4071db0 2024-03-25T19:25:26.314 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.313+0000 7fb9ec3a3640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e406dfe0 con 0x7fb9e4143f90 2024-03-25T19:25:26.314 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.313+0000 7fb9ec3a3640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:26.314 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.313+0000 7fb9ec3a3640 10 monclient(hunting): authenticate will time out at 6630.553255s 2024-03-25T19:25:26.322 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.322+0000 7fb9ea13f640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb9e40b1ed0 0x7fb9e4143880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39772/0 (socket says 172.21.15.6:39772) 2024-03-25T19:25:26.322 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.322+0000 7fb9ea13f640 1 -- 172.21.15.6:0/1256045096 learned_addr learned my addr 172.21.15.6:0/1256045096 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:25:26.338 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.339+0000 7fb9e913d640 1 -- 172.21.15.6:0/1256045096 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2848280102 0 0) 0x7fb9e406eef0 con 0x7fb9e40b1ed0 2024-03-25T19:25:26.338 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.339+0000 7fb9e913d640 10 monclient(hunting): _negotiate 2024-03-25T19:25:26.339 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.339+0000 7fb9e913d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:26.339 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.339+0000 7fb9e913d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:26.339 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.339+0000 7fb9e913d640 10 monclient(hunting): my global_id is 4734 2024-03-25T19:25:26.339 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.339+0000 7fb9e913d640 1 -- 172.21.15.6:0/1256045096 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9cc003520 con 0x7fb9e40b1ed0 2024-03-25T19:25:26.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.342+0000 7fb9e913d640 1 -- 172.21.15.6:0/1256045096 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3474015352 0 0) 0x7fb9e406f1d0 con 0x7fb9e4071db0 2024-03-25T19:25:26.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.342+0000 7fb9e913d640 10 monclient(hunting): _negotiate 2024-03-25T19:25:26.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.342+0000 7fb9e913d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:26.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.342+0000 7fb9e913d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:26.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.342+0000 7fb9e913d640 10 monclient(hunting): my global_id is 4744 2024-03-25T19:25:26.342 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.342+0000 7fb9e913d640 1 -- 172.21.15.6:0/1256045096 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9e406eef0 con 0x7fb9e4071db0 2024-03-25T19:25:26.344 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.344+0000 7fb9e913d640 1 -- 172.21.15.6:0/1256045096 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4105962717 0 0) 0x7fb9cc003520 con 0x7fb9e40b1ed0 2024-03-25T19:25:26.344 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.344+0000 7fb9e913d640 1 -- 172.21.15.6:0/1256045096 >> v1:172.21.15.6:6790/0 conn(0x7fb9e4143f90 legacy=0x7fb9e4152000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:26.344 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.344+0000 7fb9e913d640 1 -- 172.21.15.6:0/1256045096 >> v1:172.21.15.192:6789/0 conn(0x7fb9e4071db0 legacy=0x7fb9e4146740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:26.345 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.344+0000 7fb9e913d640 10 monclient: _finish_hunting 0 2024-03-25T19:25:26.345 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.344+0000 7fb9e913d640 1 monclient: found mon.noname-a 2024-03-25T19:25:26.345 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.344+0000 7fb9e913d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:26.345 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.344+0000 7fb9e913d640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-03-25T19:25:26.345 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.344+0000 7fb9e913d640 1 -- 172.21.15.6:0/1256045096 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e406e170 con 0x7fb9e40b1ed0 2024-03-25T19:25:26.345 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.344+0000 7fb9e913d640 10 monclient: _finish_auth 0 2024-03-25T19:25:26.345 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.344+0000 7fb9e913d640 10 monclient: _check_auth_tickets 2024-03-25T19:25:26.345 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.344+0000 7fb9e913d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:26.345 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.344+0000 7fb9ec3a3640 5 monclient: authenticate success, global_id 4734 2024-03-25T19:25:26.345 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.344+0000 7fb9ec3a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:26.349 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.349+0000 7fb9e913d640 1 -- 172.21.15.6:0/1256045096 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fb9d8001650 con 0x7fb9e40b1ed0 2024-03-25T19:25:26.349 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.349+0000 7fb9e913d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:26.350 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.350+0000 7fb9e913d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:25:26.350 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.350+0000 7fb9e913d640 10 monclient: dump: 2024-03-25T19:25:26.350 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:25:26.350 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:25:26.350 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:26.350 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:26.350 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:26.350 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:25:26.350 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:25:26.350 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:25:26.350 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:25:26.350 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:25:26.350 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.350+0000 7fb9e913d640 1 -- 172.21.15.6:0/1256045096 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9d8001830 con 0x7fb9e40b1ed0 2024-03-25T19:25:26.350 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.350+0000 7fb9ec3a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:26.351 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.350+0000 7fb9e913d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:26.351 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.350+0000 7fb9e913d640 1 -- 172.21.15.6:0/1256045096 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fb9d8001bc0 con 0x7fb9e40b1ed0 2024-03-25T19:25:26.351 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.350+0000 7fb9ec3a3640 10 monclient: get_monmap_and_config success 2024-03-25T19:25:26.351 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.350+0000 7fb9ec3a3640 4 set_mon_vals no callback set 2024-03-25T19:25:26.351 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.350+0000 7fb9e913d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:26.351 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.350+0000 7fb9e913d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:25:26.351 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.350+0000 7fb9e913d640 10 monclient: dump: 2024-03-25T19:25:26.351 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:25:26.351 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:25:26.351 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:26.351 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:26.351 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:26.351 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:25:26.351 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:25:26.352 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:25:26.352 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:25:26.352 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:25:26.354 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.350+0000 7fb9ec3a3640 10 monclient: shutdown 2024-03-25T19:25:26.354 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.350+0000 7fb9ec3a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:26.354 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.350+0000 7fb9ec3a3640 1 -- 172.21.15.6:0/1256045096 >> v1:172.21.15.6:6789/0 conn(0x7fb9e40b1ed0 legacy=0x7fb9e4143880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:26.354 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.350+0000 7fb9ec3a3640 1 -- 172.21.15.6:0/1256045096 shutdown_connections 2024-03-25T19:25:26.354 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.350+0000 7fb9ec3a3640 1 -- 172.21.15.6:0/1256045096 >> 172.21.15.6:0/1256045096 conn(0x7fb9e40617a0 msgr2=0x7fb9e4061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:26.354 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.351+0000 7fb9ec3a3640 1 -- 172.21.15.6:0/1256045096 shutdown_connections 2024-03-25T19:25:26.354 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.351+0000 7fb9ec3a3640 1 -- 172.21.15.6:0/1256045096 wait complete. 2024-03-25T19:25:26.354 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.352+0000 7fb9ec3a3640 10 monclient: build_initial_monmap 2024-03-25T19:25:26.354 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.352+0000 7fb9ec3a3640 10 monclient: monmap: 2024-03-25T19:25:26.354 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:25:26.354 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:25:26.354 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:25:26.354 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:25:26.354 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:26.355 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:25:26.355 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:25:26.355 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:25:26.355 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:25:26.355 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:25:26.355 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.352+0000 7fb9ec3a3640 1 librados: starting msgr at 2024-03-25T19:25:26.355 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.352+0000 7fb9ec3a3640 1 librados: starting objecter 2024-03-25T19:25:26.355 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.352+0000 7fb9ec3a3640 1 Processor -- start 2024-03-25T19:25:26.355 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.352+0000 7fb9ec3a3640 1 -- start start 2024-03-25T19:25:26.355 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.352+0000 7fb9ec3a3640 1 librados: setting wanted keys 2024-03-25T19:25:26.355 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.352+0000 7fb9ec3a3640 1 librados: calling monclient init 2024-03-25T19:25:26.355 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.352+0000 7fb9ec3a3640 10 monclient: init 2024-03-25T19:25:26.355 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.353+0000 7fb9ec3a3640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:26.356 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.353+0000 7fb9ec3a3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:25:26.356 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.353+0000 7fb9ec3a3640 10 monclient(hunting): picked mon.noname-a con 0x7fb9e4071db0 addr v1:172.21.15.6:6789/0 2024-03-25T19:25:26.356 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.353+0000 7fb9ec3a3640 10 monclient(hunting): picked mon.noname-b con 0x7fb9e40b1ed0 addr v1:172.21.15.192:6789/0 2024-03-25T19:25:26.356 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.353+0000 7fb9ec3a3640 10 monclient(hunting): picked mon.noname-c con 0x7fb9e4143f90 addr v1:172.21.15.6:6790/0 2024-03-25T19:25:26.356 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.353+0000 7fb9ec3a3640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e413aba0 con 0x7fb9e4071db0 2024-03-25T19:25:26.356 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.353+0000 7fb9ec3a3640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e4149310 con 0x7fb9e40b1ed0 2024-03-25T19:25:26.356 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.353+0000 7fb9ec3a3640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e41468d0 con 0x7fb9e4143f90 2024-03-25T19:25:26.356 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.353+0000 7fb9ec3a3640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:26.356 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.353+0000 7fb9ec3a3640 10 monclient(hunting): authenticate will time out at 6630.593196s 2024-03-25T19:25:26.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.363+0000 7fb9ea13f640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb9e4071db0 0x7fb9e41db770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39800/0 (socket says 172.21.15.6:39800) 2024-03-25T19:25:26.363 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.363+0000 7fb9ea13f640 1 -- 172.21.15.6:0/3574313574 learned_addr learned my addr 172.21.15.6:0/3574313574 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:25:26.364 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.365+0000 7fb9e2ffd640 10 client.?.objecter ms_handle_connect 0x7fb9e4071db0 2024-03-25T19:25:26.364 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.365+0000 7fb9e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:26.373 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.373+0000 7fb9e2ffd640 10 client.?.objecter ms_handle_connect 0x7fb9e40b1ed0 2024-03-25T19:25:26.373 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.373+0000 7fb9e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:26.380 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.380+0000 7fb9e2ffd640 1 -- 172.21.15.6:0/3574313574 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2607075111 0 0) 0x7fb9e413aba0 con 0x7fb9e4071db0 2024-03-25T19:25:26.380 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.380+0000 7fb9e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:25:26.380 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.380+0000 7fb9e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:26.380 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.380+0000 7fb9e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:26.380 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.380+0000 7fb9e2ffd640 10 monclient(hunting): my global_id is 4737 2024-03-25T19:25:26.380 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.380+0000 7fb9e2ffd640 1 -- 172.21.15.6:0/3574313574 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9d0004eb0 con 0x7fb9e4071db0 2024-03-25T19:25:26.382 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.383+0000 7fb9e2ffd640 10 client.?.objecter ms_handle_connect 0x7fb9e4143f90 2024-03-25T19:25:26.382 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.383+0000 7fb9e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:26.385 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.385+0000 7fb9e2ffd640 1 -- 172.21.15.6:0/3574313574 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2672729992 0 0) 0x7fb9e4149310 con 0x7fb9e40b1ed0 2024-03-25T19:25:26.385 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.385+0000 7fb9e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:25:26.385 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.385+0000 7fb9e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:26.385 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.385+0000 7fb9e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:26.385 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.385+0000 7fb9e2ffd640 10 monclient(hunting): my global_id is 4747 2024-03-25T19:25:26.386 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.385+0000 7fb9e2ffd640 1 -- 172.21.15.6:0/3574313574 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9e413aba0 con 0x7fb9e40b1ed0 2024-03-25T19:25:26.389 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.389+0000 7fb9e2ffd640 1 -- 172.21.15.6:0/3574313574 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2173793645 0 0) 0x7fb9e41468d0 con 0x7fb9e4143f90 2024-03-25T19:25:26.389 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.389+0000 7fb9e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:25:26.389 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.389+0000 7fb9e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:26.389 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.389+0000 7fb9e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:26.389 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.389+0000 7fb9e2ffd640 10 monclient(hunting): my global_id is 4736 2024-03-25T19:25:26.389 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.389+0000 7fb9e2ffd640 1 -- 172.21.15.6:0/3574313574 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9e4149310 con 0x7fb9e4143f90 2024-03-25T19:25:26.396 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.396+0000 7fb9e2ffd640 1 -- 172.21.15.6:0/3574313574 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4026719027 0 0) 0x7fb9e413aba0 con 0x7fb9e40b1ed0 2024-03-25T19:25:26.396 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.396+0000 7fb9e2ffd640 1 -- 172.21.15.6:0/3574313574 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9e41468d0 con 0x7fb9e40b1ed0 2024-03-25T19:25:26.396 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.396+0000 7fb9e2ffd640 1 -- 172.21.15.6:0/3574313574 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fb9d4001cb0 con 0x7fb9e40b1ed0 2024-03-25T19:25:26.397 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.396+0000 7fb9e2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:26.397 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.396+0000 7fb9e2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:25:26.397 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.396+0000 7fb9e2ffd640 10 monclient(hunting): dump: 2024-03-25T19:25:26.397 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:25:26.397 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:25:26.397 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:26.397 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:26.397 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:26.397 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:25:26.397 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:25:26.397 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:25:26.397 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:25:26.397 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:25:26.398 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.396+0000 7fb9e2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:25:26.407 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.406+0000 7fb9e2ffd640 1 -- 172.21.15.6:0/3574313574 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3958591155 0 0) 0x7fb9e41468d0 con 0x7fb9e40b1ed0 2024-03-25T19:25:26.407 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.406+0000 7fb9e2ffd640 1 -- 172.21.15.6:0/3574313574 >> v1:172.21.15.6:6790/0 conn(0x7fb9e4143f90 legacy=0x7fb9e413a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:26.407 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.406+0000 7fb9e2ffd640 1 -- 172.21.15.6:0/3574313574 >> v1:172.21.15.6:6789/0 conn(0x7fb9e4071db0 legacy=0x7fb9e41db770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:26.407 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.406+0000 7fb9e2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:25:26.407 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.406+0000 7fb9e2ffd640 1 monclient: found mon.b 2024-03-25T19:25:26.407 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.406+0000 7fb9e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:26.407 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.406+0000 7fb9e2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:25:26.407 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.406+0000 7fb9e2ffd640 1 -- 172.21.15.6:0/3574313574 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e41d92b0 con 0x7fb9e40b1ed0 2024-03-25T19:25:26.407 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.406+0000 7fb9e2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:25:26.407 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.406+0000 7fb9e2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:25:26.407 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.406+0000 7fb9e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:26.407 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.406+0000 7fb9ec3a3640 5 monclient: authenticate success, global_id 4747 2024-03-25T19:25:26.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.406+0000 7fb9ec3a3640 10 monclient: _renew_subs 2024-03-25T19:25:26.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.406+0000 7fb9ec3a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:25:26.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.406+0000 7fb9ec3a3640 1 -- 172.21.15.6:0/3574313574 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9e41dbe80 con 0x7fb9e40b1ed0 2024-03-25T19:25:26.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.407+0000 7fb9ec3a3640 10 client.4747.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:25:26.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.407+0000 7fb9ec3a3640 10 monclient: _renew_subs 2024-03-25T19:25:26.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.407+0000 7fb9ec3a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:25:26.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.407+0000 7fb9ec3a3640 1 -- 172.21.15.6:0/3574313574 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9e41499f0 con 0x7fb9e40b1ed0 2024-03-25T19:25:26.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.407+0000 7fb9ec3a3640 1 librados: init done 2024-03-25T19:25:26.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.408+0000 7fb9ec3a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:26.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.408+0000 7fb9ec3a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:26.409 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.408+0000 7fb9ec3a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:25:26.409 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.408+0000 7fb9ec3a3640 1 -- 172.21.15.6:0/3574313574 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9e406dbd0 con 0x7fb9e40b1ed0 2024-03-25T19:25:26.687 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.687+0000 7fb9e2ffd640 1 -- 172.21.15.6:0/3574313574 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9d4003990 con 0x7fb9e40b1ed0 2024-03-25T19:25:26.687 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.687+0000 7fb9e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:26.687 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.687+0000 7fb9e2ffd640 1 -- 172.21.15.6:0/3574313574 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fb9d4003ce0 con 0x7fb9e40b1ed0 2024-03-25T19:25:26.687 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.687+0000 7fb9e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:26.687 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.687+0000 7fb9e2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:25:26.687 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.687+0000 7fb9e2ffd640 10 monclient: dump: 2024-03-25T19:25:26.687 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:25:26.687 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:25:26.687 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:26.688 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:26.688 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:26.688 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:25:26.688 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:25:26.688 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:25:26.688 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:25:26.688 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:25:26.688 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.687+0000 7fb9e893c640 4 set_mon_vals no callback set 2024-03-25T19:25:26.689 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.687+0000 7fb9e2ffd640 1 -- 172.21.15.6:0/3574313574 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7fb9d4003f00 con 0x7fb9e40b1ed0 2024-03-25T19:25:26.689 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.688+0000 7fb9ea13f640 10 monclient: get_auth_request con 0x7fb9d006dee0 auth_method 0 2024-03-25T19:25:26.689 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.688+0000 7fb9e2ffd640 10 client.4747.objecter ms_handle_connect 0x7fb9d006dee0 2024-03-25T19:25:26.692 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.692+0000 7fb9e2ffd640 1 -- 172.21.15.6:0/3574313574 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3970010106 0 0) 0x7fb9d4053960 con 0x7fb9e40b1ed0 2024-03-25T19:25:26.692 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.692+0000 7fb9e2ffd640 10 client.4747.objecter ms_dispatch 0x7fb9e4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:25:26.692 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.692+0000 7fb9e2ffd640 3 client.4747.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:25:26.692 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.692+0000 7fb9e2ffd640 3 client.4747.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:25:26.693 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.692+0000 7fb9e2ffd640 20 client.4747.objecter dump_active .. 0 homeless 2024-03-25T19:25:26.693 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.694+0000 7fb9e2ffd640 1 -- 172.21.15.6:0/3574313574 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb9d404f470 con 0x7fb9e40b1ed0 2024-03-25T19:25:26.693 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.694+0000 7fb9e2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:26.693 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.694+0000 7fb9e2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:25:26.940 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.940+0000 7fb9ec3a3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:26.940 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.940+0000 7fb9ec3a3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:26.940 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.940+0000 7fb9ec3a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:25:26.940 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.940+0000 7fb9ec3a3640 1 -- 172.21.15.6:0/3574313574 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9e40688c0 con 0x7fb9e40b1ed0 2024-03-25T19:25:26.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.958+0000 7fb9e2ffd640 1 -- 172.21.15.6:0/3574313574 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3097771541) 0x7fb9d40513a0 con 0x7fb9e40b1ed0 2024-03-25T19:25:26.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.958+0000 7fb9e2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:26.958 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.958+0000 7fb9e2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:25:26.958 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:25:26.958 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":11,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:24:09.307681+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:24:09.307681+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:25:26.960 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.960+0000 7fb9ec3a3640 10 librados: watch_flush enter 2024-03-25T19:25:26.960 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.960+0000 7fb9ec3a3640 10 librados: watch_flush exit 2024-03-25T19:25:26.961 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.960+0000 7fb9ec3a3640 20 client.4747.objecter shutdown clearing up homeless session... 2024-03-25T19:25:26.961 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.960+0000 7fb9ec3a3640 10 client.4747.objecter successfully canceled tick 2024-03-25T19:25:26.961 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.960+0000 7fb9ec3a3640 1 -- 172.21.15.6:0/3574313574 >> v1:172.21.15.6:6800/2936652650 conn(0x7fb9d006dee0 legacy=0x7fb9d00703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:26.961 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.960+0000 7fb9ec3a3640 10 monclient: shutdown 2024-03-25T19:25:26.961 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.960+0000 7fb9ec3a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:26.961 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.960+0000 7fb9ec3a3640 1 -- 172.21.15.6:0/3574313574 >> v1:172.21.15.192:6789/0 conn(0x7fb9e40b1ed0 legacy=0x7fb9e4139d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:26.961 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.960+0000 7fb9ec3a3640 1 -- 172.21.15.6:0/3574313574 shutdown_connections 2024-03-25T19:25:26.961 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.960+0000 7fb9ec3a3640 1 -- 172.21.15.6:0/3574313574 >> 172.21.15.6:0/3574313574 conn(0x7fb9e40617a0 msgr2=0x7fb9e4143650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:26.961 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.960+0000 7fb9ec3a3640 1 -- 172.21.15.6:0/3574313574 shutdown_connections 2024-03-25T19:25:26.961 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.960+0000 7fb9ec3a3640 1 -- 172.21.15.6:0/3574313574 wait complete. 2024-03-25T19:25:26.962 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:26.960+0000 7fb9ec3a3640 1 librados: shutdown 2024-03-25T19:25:26.971 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:25:29.668 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:11:47.014 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:29.676 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:11:47.023 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:29.715 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:11:47.062 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:29.732 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:11:47.050 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:29.736 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:11:47.083 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:29.736 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:11:47.054 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:29.741 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:11:47.088 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:29.775 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:11:47.093 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:29.783 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:11:47.130 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:33.778 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:25:33.894 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.893+0000 7fa51fd82640 10 monclient: get_monmap_and_config 2024-03-25T19:25:33.894 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.893+0000 7fa51fd82640 10 monclient: build_initial_monmap 2024-03-25T19:25:33.894 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.893+0000 7fa51fd82640 10 monclient: monmap: 2024-03-25T19:25:33.894 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:25:33.894 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:25:33.894 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:25:33.894950+0000 2024-03-25T19:25:33.894 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:25:33.894950+0000 2024-03-25T19:25:33.894 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:33.894 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:25:33.894 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:25:33.894 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-03-25T19:25:33.895 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-03-25T19:25:33.895 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:25:33.895 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.894+0000 7fa51fd82640 1 Processor -- start 2024-03-25T19:25:33.896 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.894+0000 7fa51fd82640 1 -- start start 2024-03-25T19:25:33.896 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.894+0000 7fa51fd82640 10 monclient: init 2024-03-25T19:25:33.896 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.895+0000 7fa51fd82640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:33.896 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.895+0000 7fa51fd82640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:25:33.896 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.895+0000 7fa51fd82640 10 monclient(hunting): picked mon.noname-a con 0x7fa5180b4e70 addr v1:172.21.15.6:6789/0 2024-03-25T19:25:33.896 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.895+0000 7fa51fd82640 10 monclient(hunting): picked mon.noname-c con 0x7fa5180b44b0 addr v1:172.21.15.192:6789/0 2024-03-25T19:25:33.896 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.895+0000 7fa51fd82640 10 monclient(hunting): picked mon.noname-b con 0x7fa5180b1b00 addr v1:172.21.15.6:6790/0 2024-03-25T19:25:33.896 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.895+0000 7fa51fd82640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5180a96f0 con 0x7fa5180b4e70 2024-03-25T19:25:33.896 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.895+0000 7fa51fd82640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5180ab0e0 con 0x7fa5180b44b0 2024-03-25T19:25:33.896 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.895+0000 7fa51fd82640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa518145f70 con 0x7fa5180b1b00 2024-03-25T19:25:33.896 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.895+0000 7fa51fd82640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:33.896 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.895+0000 7fa51fd82640 10 monclient(hunting): authenticate will time out at 6638.135150s 2024-03-25T19:25:33.897 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.895+0000 7fa51e31f640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fa5180b1b00 0x7fa51814dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:55820/0 (socket says 172.21.15.6:55820) 2024-03-25T19:25:33.897 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.895+0000 7fa51e31f640 1 -- 172.21.15.6:0/195648619 learned_addr learned my addr 172.21.15.6:0/195648619 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:25:33.942 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.942+0000 7fa51cb1c640 1 -- 172.21.15.6:0/195648619 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 886663137 0 0) 0x7fa5180a96f0 con 0x7fa5180b4e70 2024-03-25T19:25:33.942 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.942+0000 7fa51cb1c640 10 monclient(hunting): _negotiate 2024-03-25T19:25:33.943 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.942+0000 7fa51cb1c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:33.943 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.942+0000 7fa51cb1c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:33.943 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.942+0000 7fa51cb1c640 10 monclient(hunting): my global_id is 4740 2024-03-25T19:25:33.943 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.942+0000 7fa51cb1c640 1 -- 172.21.15.6:0/195648619 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa500003520 con 0x7fa5180b4e70 2024-03-25T19:25:33.959 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.958+0000 7fa51cb1c640 1 -- 172.21.15.6:0/195648619 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3293971959 0 0) 0x7fa500003520 con 0x7fa5180b4e70 2024-03-25T19:25:33.959 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.958+0000 7fa51cb1c640 1 -- 172.21.15.6:0/195648619 >> v1:172.21.15.6:6790/0 conn(0x7fa5180b1b00 legacy=0x7fa51814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:33.959 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.958+0000 7fa51cb1c640 1 -- 172.21.15.6:0/195648619 >> v1:172.21.15.192:6789/0 conn(0x7fa5180b44b0 legacy=0x7fa518145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:33.959 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.958+0000 7fa51cb1c640 10 monclient: _finish_hunting 0 2024-03-25T19:25:33.959 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.958+0000 7fa51cb1c640 1 monclient: found mon.noname-a 2024-03-25T19:25:33.959 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.958+0000 7fa51cb1c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:33.959 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.958+0000 7fa51cb1c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-03-25T19:25:33.959 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.958+0000 7fa51cb1c640 1 -- 172.21.15.6:0/195648619 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5180a9b80 con 0x7fa5180b4e70 2024-03-25T19:25:33.959 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.958+0000 7fa51cb1c640 10 monclient: _finish_auth 0 2024-03-25T19:25:33.959 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.958+0000 7fa51cb1c640 10 monclient: _check_auth_tickets 2024-03-25T19:25:33.960 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.958+0000 7fa51cb1c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:33.960 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.958+0000 7fa51cb1c640 1 -- 172.21.15.6:0/195648619 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fa510001690 con 0x7fa5180b4e70 2024-03-25T19:25:33.960 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.958+0000 7fa51fd82640 5 monclient: authenticate success, global_id 4740 2024-03-25T19:25:33.960 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.958+0000 7fa51fd82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:33.960 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.958+0000 7fa51cb1c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:33.960 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.958+0000 7fa51cb1c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:25:33.960 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.958+0000 7fa51cb1c640 10 monclient: dump: 2024-03-25T19:25:33.960 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:25:33.960 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:25:33.960 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:33.960 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:33.960 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:33.960 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:25:33.960 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:25:33.961 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:25:33.961 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:25:33.961 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:25:33.961 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.958+0000 7fa51fd82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:33.970 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.970+0000 7fa51cb1c640 1 -- 172.21.15.6:0/195648619 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5100017f0 con 0x7fa5180b4e70 2024-03-25T19:25:33.970 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.970+0000 7fa51cb1c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:33.970 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.970+0000 7fa51cb1c640 1 -- 172.21.15.6:0/195648619 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fa510001c00 con 0x7fa5180b4e70 2024-03-25T19:25:33.970 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.970+0000 7fa51fd82640 10 monclient: get_monmap_and_config success 2024-03-25T19:25:33.970 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.970+0000 7fa51fd82640 4 set_mon_vals no callback set 2024-03-25T19:25:33.970 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.970+0000 7fa51cb1c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:33.970 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.970+0000 7fa51cb1c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:25:33.970 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.970+0000 7fa51cb1c640 10 monclient: dump: 2024-03-25T19:25:33.971 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:25:33.971 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:25:33.971 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:33.971 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:33.971 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:33.971 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:25:33.971 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:25:33.971 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:25:33.971 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:25:33.971 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:25:33.973 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.971+0000 7fa51fd82640 10 monclient: shutdown 2024-03-25T19:25:33.973 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.971+0000 7fa51fd82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:33.973 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.971+0000 7fa51fd82640 1 -- 172.21.15.6:0/195648619 >> v1:172.21.15.6:6789/0 conn(0x7fa5180b4e70 legacy=0x7fa51813f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:33.973 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.971+0000 7fa51fd82640 1 -- 172.21.15.6:0/195648619 shutdown_connections 2024-03-25T19:25:33.973 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.971+0000 7fa51fd82640 1 -- 172.21.15.6:0/195648619 >> 172.21.15.6:0/195648619 conn(0x7fa5180617a0 msgr2=0x7fa518061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:33.973 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.971+0000 7fa51fd82640 1 -- 172.21.15.6:0/195648619 shutdown_connections 2024-03-25T19:25:33.973 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.971+0000 7fa51fd82640 1 -- 172.21.15.6:0/195648619 wait complete. 2024-03-25T19:25:33.973 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.972+0000 7fa51fd82640 10 monclient: build_initial_monmap 2024-03-25T19:25:33.973 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.972+0000 7fa51fd82640 10 monclient: monmap: 2024-03-25T19:25:33.973 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-03-25T19:25:33.973 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:25:33.973 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-03-25T19:25:33.974 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-03-25T19:25:33.974 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:33.974 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-03-25T19:25:33.974 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-03-25T19:25:33.974 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-03-25T19:25:33.974 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-03-25T19:25:33.974 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:25:33.974 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.972+0000 7fa51fd82640 1 librados: starting msgr at 2024-03-25T19:25:33.974 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.973+0000 7fa51fd82640 1 librados: starting objecter 2024-03-25T19:25:33.974 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.973+0000 7fa51fd82640 1 Processor -- start 2024-03-25T19:25:33.974 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.973+0000 7fa51fd82640 1 -- start start 2024-03-25T19:25:33.974 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.973+0000 7fa51fd82640 1 librados: setting wanted keys 2024-03-25T19:25:33.974 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.973+0000 7fa51fd82640 1 librados: calling monclient init 2024-03-25T19:25:33.975 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.973+0000 7fa51fd82640 10 monclient: init 2024-03-25T19:25:33.975 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.974+0000 7fa51fd82640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:33.976 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.974+0000 7fa51fd82640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:25:33.976 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.974+0000 7fa51fd82640 10 monclient(hunting): picked mon.noname-a con 0x7fa5180b1b00 addr v1:172.21.15.6:6789/0 2024-03-25T19:25:33.976 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.974+0000 7fa51fd82640 10 monclient(hunting): picked mon.noname-c con 0x7fa5180b44b0 addr v1:172.21.15.6:6790/0 2024-03-25T19:25:33.976 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.974+0000 7fa51fd82640 10 monclient(hunting): picked mon.noname-b con 0x7fa5180b4e70 addr v1:172.21.15.192:6789/0 2024-03-25T19:25:33.976 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.974+0000 7fa51fd82640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa518138600 con 0x7fa5180b1b00 2024-03-25T19:25:33.976 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.974+0000 7fa51fd82640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa518140160 con 0x7fa5180b4e70 2024-03-25T19:25:33.976 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.974+0000 7fa51fd82640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5181387b0 con 0x7fa5180b44b0 2024-03-25T19:25:33.976 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.974+0000 7fa51fd82640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:33.976 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.974+0000 7fa51fd82640 10 monclient(hunting): authenticate will time out at 6638.213911s 2024-03-25T19:25:33.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.988+0000 7fa51e31f640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fa5180b4e70 0x7fa5181d75e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.6:52980/0 (socket says 172.21.15.6:52980) 2024-03-25T19:25:33.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.988+0000 7fa51e31f640 1 -- 172.21.15.6:0/3284989278 learned_addr learned my addr 172.21.15.6:0/3284989278 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-25T19:25:33.989 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.990+0000 7fa5167fc640 10 client.?.objecter ms_handle_connect 0x7fa5180b4e70 2024-03-25T19:25:33.990 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.990+0000 7fa5167fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:33.990 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.990+0000 7fa5167fc640 10 client.?.objecter ms_handle_connect 0x7fa5180b1b00 2024-03-25T19:25:33.990 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.990+0000 7fa5167fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:33.994 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.994+0000 7fa5167fc640 1 -- 172.21.15.6:0/3284989278 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1543287004 0 0) 0x7fa518140160 con 0x7fa5180b4e70 2024-03-25T19:25:33.994 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.994+0000 7fa5167fc640 10 monclient(hunting): _negotiate 2024-03-25T19:25:33.994 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.994+0000 7fa5167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:33.994 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.994+0000 7fa5167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:33.994 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.994+0000 7fa5167fc640 10 monclient(hunting): my global_id is 4753 2024-03-25T19:25:33.994 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:33.994+0000 7fa5167fc640 1 -- 172.21.15.6:0/3284989278 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa504004eb0 con 0x7fa5180b4e70 2024-03-25T19:25:34.008 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.007+0000 7fa5167fc640 1 -- 172.21.15.6:0/3284989278 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 38573810 0 0) 0x7fa504004eb0 con 0x7fa5180b4e70 2024-03-25T19:25:34.008 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.008+0000 7fa5167fc640 1 -- 172.21.15.6:0/3284989278 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa518140160 con 0x7fa5180b4e70 2024-03-25T19:25:34.008 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.008+0000 7fa5167fc640 1 -- 172.21.15.6:0/3284989278 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fa508001bb0 con 0x7fa5180b4e70 2024-03-25T19:25:34.008 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.008+0000 7fa5167fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:34.008 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.008+0000 7fa5167fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:25:34.008 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.008+0000 7fa5167fc640 10 monclient(hunting): dump: 2024-03-25T19:25:34.008 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:25:34.008 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:25:34.008 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:34.008 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:34.008 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:34.008 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:25:34.008 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:25:34.009 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:25:34.009 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:25:34.009 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:25:34.009 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.008+0000 7fa5167fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:25:34.009 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.010+0000 7fa5167fc640 10 client.?.objecter ms_handle_connect 0x7fa5180b44b0 2024-03-25T19:25:34.009 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.010+0000 7fa5167fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:34.012 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.012+0000 7fa5167fc640 1 -- 172.21.15.6:0/3284989278 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2879870687 0 0) 0x7fa518138600 con 0x7fa5180b1b00 2024-03-25T19:25:34.012 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.012+0000 7fa5167fc640 10 monclient(hunting): _negotiate 2024-03-25T19:25:34.012 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.012+0000 7fa5167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:34.012 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.012+0000 7fa5167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:34.012 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.012+0000 7fa5167fc640 10 monclient(hunting): my global_id is 4743 2024-03-25T19:25:34.012 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.012+0000 7fa5167fc640 1 -- 172.21.15.6:0/3284989278 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa504004eb0 con 0x7fa5180b1b00 2024-03-25T19:25:34.017 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.017+0000 7fa5167fc640 1 -- 172.21.15.6:0/3284989278 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2111485153 0 0) 0x7fa518140160 con 0x7fa5180b4e70 2024-03-25T19:25:34.017 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.017+0000 7fa5167fc640 1 -- 172.21.15.6:0/3284989278 >> v1:172.21.15.6:6790/0 conn(0x7fa5180b44b0 legacy=0x7fa518130e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:34.017 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.017+0000 7fa5167fc640 1 -- 172.21.15.6:0/3284989278 >> v1:172.21.15.6:6789/0 conn(0x7fa5180b1b00 legacy=0x7fa518137ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:34.017 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.017+0000 7fa5167fc640 10 monclient: _finish_hunting 0 2024-03-25T19:25:34.017 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.017+0000 7fa5167fc640 1 monclient: found mon.b 2024-03-25T19:25:34.017 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.017+0000 7fa5167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:34.017 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.017+0000 7fa5167fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:25:34.018 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.017+0000 7fa5167fc640 1 -- 172.21.15.6:0/3284989278 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa518138940 con 0x7fa5180b4e70 2024-03-25T19:25:34.018 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.017+0000 7fa5167fc640 10 monclient: _finish_auth 0 2024-03-25T19:25:34.018 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.017+0000 7fa5167fc640 10 monclient: _check_auth_tickets 2024-03-25T19:25:34.018 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.017+0000 7fa5167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:34.018 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.017+0000 7fa51fd82640 5 monclient: authenticate success, global_id 4753 2024-03-25T19:25:34.018 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.017+0000 7fa51fd82640 10 monclient: _renew_subs 2024-03-25T19:25:34.018 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.017+0000 7fa51fd82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:25:34.018 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.017+0000 7fa51fd82640 1 -- 172.21.15.6:0/3284989278 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa518131560 con 0x7fa5180b4e70 2024-03-25T19:25:34.018 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.017+0000 7fa51fd82640 10 client.4753.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:25:34.018 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.017+0000 7fa51fd82640 10 monclient: _renew_subs 2024-03-25T19:25:34.018 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.017+0000 7fa51fd82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:25:34.018 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.017+0000 7fa51fd82640 1 -- 172.21.15.6:0/3284989278 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5181d7cf0 con 0x7fa5180b4e70 2024-03-25T19:25:34.018 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.017+0000 7fa51fd82640 1 librados: init done 2024-03-25T19:25:34.019 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.018+0000 7fa51fd82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:34.019 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.018+0000 7fa51fd82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:34.019 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.018+0000 7fa51fd82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:25:34.019 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.018+0000 7fa51fd82640 1 -- 172.21.15.6:0/3284989278 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa51804f730 con 0x7fa5180b4e70 2024-03-25T19:25:34.293 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.293+0000 7fa5167fc640 1 -- 172.21.15.6:0/3284989278 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa508004b60 con 0x7fa5180b4e70 2024-03-25T19:25:34.293 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.293+0000 7fa5167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:34.293 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.293+0000 7fa5167fc640 1 -- 172.21.15.6:0/3284989278 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3948913340 0 0) 0x7fa508004f70 con 0x7fa5180b4e70 2024-03-25T19:25:34.293 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.293+0000 7fa5167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:34.293 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.293+0000 7fa5167fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:25:34.293 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.293+0000 7fa5167fc640 10 monclient: dump: 2024-03-25T19:25:34.293 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-03-25T19:25:34.293 INFO:teuthology.misc.health.smithi006.stderr:fsid 673040aa-a547-4a0b-a8e8-76613d195b59 2024-03-25T19:25:34.293 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:34.293 INFO:teuthology.misc.health.smithi006.stderr:created 2024-03-25T19:13:09.666361+0000 2024-03-25T19:25:34.293 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:34.294 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-03-25T19:25:34.294 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-03-25T19:25:34.294 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-03-25T19:25:34.294 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-03-25T19:25:34.294 INFO:teuthology.misc.health.smithi006.stderr: 2024-03-25T19:25:34.294 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.293+0000 7fa517fff640 4 set_mon_vals no callback set 2024-03-25T19:25:34.294 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.294+0000 7fa5167fc640 1 -- 172.21.15.6:0/3284989278 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3216246878 0 0) 0x7fa508005190 con 0x7fa5180b4e70 2024-03-25T19:25:34.295 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.294+0000 7fa51db1e640 10 monclient: get_auth_request con 0x7fa50406dd90 auth_method 0 2024-03-25T19:25:34.295 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.294+0000 7fa5167fc640 10 client.4753.objecter ms_handle_connect 0x7fa50406dd90 2024-03-25T19:25:34.299 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.299+0000 7fa5167fc640 1 -- 172.21.15.6:0/3284989278 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3970010106 0 0) 0x7fa508054a20 con 0x7fa5180b4e70 2024-03-25T19:25:34.300 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.299+0000 7fa5167fc640 10 client.4753.objecter ms_dispatch 0x7fa518000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:25:34.300 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.299+0000 7fa5167fc640 3 client.4753.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:25:34.300 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.299+0000 7fa5167fc640 3 client.4753.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:25:34.300 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.300+0000 7fa5167fc640 20 client.4753.objecter dump_active .. 0 homeless 2024-03-25T19:25:34.300 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.300+0000 7fa5167fc640 1 -- 172.21.15.6:0/3284989278 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa508050530 con 0x7fa5180b4e70 2024-03-25T19:25:34.300 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.301+0000 7fa5167fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:34.300 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.301+0000 7fa5167fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:25:34.554 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.554+0000 7fa51fd82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:34.554 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.554+0000 7fa51fd82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:34.554 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.554+0000 7fa51fd82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-03-25T19:25:34.554 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.554+0000 7fa51fd82640 1 -- 172.21.15.6:0/3284989278 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5180aa090 con 0x7fa5180b4e70 2024-03-25T19:25:34.576 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.576+0000 7fa5167fc640 1 -- 172.21.15.6:0/3284989278 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3097771541) 0x7fa508052460 con 0x7fa5180b4e70 2024-03-25T19:25:34.576 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.576+0000 7fa5167fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:34.576 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.576+0000 7fa5167fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:25:34.577 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-25T19:25:34.577 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":11,"fsid":"673040aa-a547-4a0b-a8e8-76613d195b59","created":"2024-03-25T19:14:05.093147+0000","modified":"2024-03-25T19:24:09.307681+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:24:09.307681+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15732146-8cac-4747-96aa-c2cad384c361","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"2a83e0b2-dbc3-40be-8026-48e14034e957","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3072cff1-2822-4922-9ec1-97b6e8656a3a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d14827e0-9625-4ef5-8adf-d3430bde36e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1937922-f839-4d21-bd5b-f460adafe633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31abd8d9-93e5-4f4b-928e-d044c4b46999","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"861c9465-28f4-4980-9f1b-e411962ed333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"14bd0271-d543-4ea4-ab4c-61475195fc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:25:34.579 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.579+0000 7fa51fd82640 10 librados: watch_flush enter 2024-03-25T19:25:34.580 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.579+0000 7fa51fd82640 10 librados: watch_flush exit 2024-03-25T19:25:34.580 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.579+0000 7fa51fd82640 20 client.4753.objecter shutdown clearing up homeless session... 2024-03-25T19:25:34.580 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.579+0000 7fa51fd82640 10 client.4753.objecter successfully canceled tick 2024-03-25T19:25:34.580 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.579+0000 7fa51fd82640 1 -- 172.21.15.6:0/3284989278 >> v1:172.21.15.6:6800/2936652650 conn(0x7fa50406dd90 legacy=0x7fa504070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:34.580 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.579+0000 7fa51fd82640 10 monclient: shutdown 2024-03-25T19:25:34.580 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.579+0000 7fa51fd82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:34.580 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.579+0000 7fa51fd82640 1 -- 172.21.15.6:0/3284989278 >> v1:172.21.15.192:6789/0 conn(0x7fa5180b4e70 legacy=0x7fa5181d75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:34.580 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.579+0000 7fa51fd82640 1 -- 172.21.15.6:0/3284989278 shutdown_connections 2024-03-25T19:25:34.580 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.579+0000 7fa51fd82640 1 -- 172.21.15.6:0/3284989278 >> 172.21.15.6:0/3284989278 conn(0x7fa5180617a0 msgr2=0x7fa5180b6720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:34.580 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.579+0000 7fa51fd82640 1 -- 172.21.15.6:0/3284989278 shutdown_connections 2024-03-25T19:25:34.580 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.579+0000 7fa51fd82640 1 -- 172.21.15.6:0/3284989278 wait complete. 2024-03-25T19:25:34.580 INFO:teuthology.misc.health.smithi006.stderr:2024-03-25T19:25:34.579+0000 7fa51fd82640 1 librados: shutdown 2024-03-25T19:25:34.590 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:25:34.591 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T19:25:34.592 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T19:25:34.593 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T19:25:34.593 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:25:36.668 INFO:tasks.ceph.osd.0.smithi006.stderr:==00:00:11:26.160 35451== 2024-03-25T19:25:36.668 INFO:tasks.ceph.osd.0.smithi006.stderr:==00:00:11:26.160 35451== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:25:36.708 INFO:tasks.ceph.osd.0.smithi006.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:25:40.694 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:25:40.695 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi006 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T19:25:40.696 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T19:25:40.696 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T19:25:40.696 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:25:42.708 INFO:tasks.ceph.osd.1.smithi006.stderr:==00:00:11:32.176 35454== 2024-03-25T19:25:42.708 INFO:tasks.ceph.osd.1.smithi006.stderr:==00:00:11:32.176 35454== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:25:42.749 INFO:tasks.ceph.osd.1.smithi006.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:25:46.797 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:25:46.798 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi006 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T19:25:46.798 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T19:25:46.798 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T19:25:46.798 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:25:48.799 INFO:tasks.ceph.osd.2.smithi006.stderr:==00:00:11:38.283 35452== 2024-03-25T19:25:48.799 INFO:tasks.ceph.osd.2.smithi006.stderr:==00:00:11:38.283 35452== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:25:48.837 INFO:tasks.ceph.osd.2.smithi006.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:25:52.900 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:25:52.900 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi006 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T19:25:52.900 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T19:25:52.901 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T19:25:52.901 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:25:54.671 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:12:12.017 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:54.713 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:12:12.031 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:54.715 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:12:12.033 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:54.721 INFO:tasks.ceph.osd.3.smithi006.stderr:==00:00:11:44.190 35453== 2024-03-25T19:25:54.721 INFO:tasks.ceph.osd.3.smithi006.stderr:==00:00:11:44.190 35453== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:25:54.725 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:12:12.043 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:54.734 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:12:12.081 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:54.756 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:12:12.103 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:54.764 INFO:tasks.ceph.osd.3.smithi006.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:25:54.787 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:12:12.134 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:54.793 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:12:12.140 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:54.819 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:12:12.166 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:59.002 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:25:59.003 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi006 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T19:25:59.003 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T19:25:59.003 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T19:25:59.003 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:26:01.093 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:11:50.558 35211== 2024-03-25T19:26:01.093 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:11:50.558 35211== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:26:01.136 INFO:tasks.ceph.osd.4.smithi192.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:26:05.105 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:26:05.106 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi192 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T19:26:05.106 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T19:26:05.106 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T19:26:05.106 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:26:06.913 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:11:56.394 35208== 2024-03-25T19:26:06.913 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:11:56.394 35208== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:26:06.949 INFO:tasks.ceph.osd.5.smithi192.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:26:11.208 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:26:11.209 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi192 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T19:26:11.209 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T19:26:11.209 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T19:26:11.210 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:26:12.940 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:12:02.421 35209== 2024-03-25T19:26:12.941 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:12:02.421 35209== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:26:12.982 INFO:tasks.ceph.osd.6.smithi192.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:26:14.750 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:12:32.096 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:14.754 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:12:32.101 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:14.775 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:12:32.122 35137== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:14.801 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:12:32.119 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:14.805 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:12:32.124 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:14.806 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:12:32.152 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:14.809 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:12:32.156 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:14.818 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:12:32.136 34665== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:14.823 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:12:32.170 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:17.312 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:26:17.312 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi192 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T19:26:17.312 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T19:26:17.313 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T19:26:17.313 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:26:18.854 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:12:08.333 35210== 2024-03-25T19:26:18.854 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:12:08.333 35210== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:26:18.884 INFO:tasks.ceph.osd.7.smithi192.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:26:23.415 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:26:23.415 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi192 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T19:26:23.416 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T19:26:23.416 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T19:26:23.416 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T19:26:23.416 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:26:23.445 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T19:26:23.445 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T19:26:23.445 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:26:23.472 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T19:26:23.473 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T19:26:23.473 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T19:26:23.473 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:26:23.503 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-03-25T19:26:23.490+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35104) UID: 0 2024-03-25T19:26:23.504 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-03-25T19:26:23.492+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T19:28:08.021 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:14:25.367 35137== 2024-03-25T19:28:08.022 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:14:25.367 35137== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:28:08.030 INFO:tasks.ceph.mon.a.smithi006.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:28:13.299 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:28:13.299 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi006 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T19:28:13.300 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T19:28:13.300 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T19:28:13.300 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:28:13.321 INFO:tasks.ceph.mon.c.smithi006.stderr:2024-03-25T19:28:13.305+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 35106) UID: 0 2024-03-25T19:28:13.321 INFO:tasks.ceph.mon.c.smithi006.stderr:2024-03-25T19:28:13.307+0000 9f86640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-25T19:28:13.851 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:14:31.198 35140== 2024-03-25T19:28:13.851 INFO:tasks.ceph.mon.c.smithi006.stderr:==00:00:14:31.198 35140== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:28:13.861 INFO:tasks.ceph.mon.c.smithi006.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:28:19.402 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:28:19.403 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi006 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T19:28:19.403 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T19:28:19.403 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T19:28:19.403 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:28:19.419 INFO:tasks.ceph.mon.b.smithi192.stderr:2024-03-25T19:28:19.408+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 34661) UID: 0 2024-03-25T19:28:19.420 INFO:tasks.ceph.mon.b.smithi192.stderr:2024-03-25T19:28:19.410+0000 9f86640 -1 mon.b@1(leader) e1 *** Got Signal Terminated *** 2024-03-25T19:29:02.052 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:19.369 34665== 2024-03-25T19:29:02.052 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:19.369 34665== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:29:02.061 INFO:tasks.ceph.mon.b.smithi192.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:29:02.113 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:29:02.113 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi192 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T19:29:02.114 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T19:29:02.114 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T19:29:02.115 DEBUG:teuthology.orchestra.run.smithi006:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T19:29:02.160 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi006.front.sepia.ceph.com 2024-03-25T19:29:02.160 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T19:29:02.220 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi006.front.sepia.ceph.com 2024-03-25T19:29:02.220 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T19:29:02.303 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi006.front.sepia.ceph.com 2024-03-25T19:29:02.303 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T19:29:02.361 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi006.front.sepia.ceph.com 2024-03-25T19:29:02.361 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T19:29:02.519 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi192.front.sepia.ceph.com 2024-03-25T19:29:02.519 DEBUG:teuthology.orchestra.run.smithi192:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T19:29:02.639 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi192.front.sepia.ceph.com 2024-03-25T19:29:02.639 DEBUG:teuthology.orchestra.run.smithi192:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T19:29:02.706 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi192.front.sepia.ceph.com 2024-03-25T19:29:02.706 DEBUG:teuthology.orchestra.run.smithi192:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T19:29:02.773 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi192.front.sepia.ceph.com 2024-03-25T19:29:02.773 DEBUG:teuthology.orchestra.run.smithi192:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T19:29:02.828 INFO:tasks.ceph:Archiving mon data... 2024-03-25T19:29:02.829 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621303/data/mon.a.tgz 2024-03-25T19:29:02.830 DEBUG:teuthology.orchestra.run.smithi006:> mktemp 2024-03-25T19:29:02.846 INFO:teuthology.orchestra.run.smithi006.stdout:/tmp/tmp.VUjNkT1OSJ 2024-03-25T19:29:02.846 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.VUjNkT1OSJ 2024-03-25T19:29:02.945 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0666 /tmp/tmp.VUjNkT1OSJ 2024-03-25T19:29:03.031 DEBUG:teuthology.orchestra.remote:smithi006:/tmp/tmp.VUjNkT1OSJ is 134KB 2024-03-25T19:29:03.083 DEBUG:teuthology.orchestra.run.smithi006:> rm -fr /tmp/tmp.VUjNkT1OSJ 2024-03-25T19:29:03.098 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621303/data/mon.c.tgz 2024-03-25T19:29:03.098 DEBUG:teuthology.orchestra.run.smithi006:> mktemp 2024-03-25T19:29:03.152 INFO:teuthology.orchestra.run.smithi006.stdout:/tmp/tmp.qmRABefvBd 2024-03-25T19:29:03.152 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.qmRABefvBd 2024-03-25T19:29:03.246 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0666 /tmp/tmp.qmRABefvBd 2024-03-25T19:29:03.330 DEBUG:teuthology.orchestra.remote:smithi006:/tmp/tmp.qmRABefvBd is 164KB 2024-03-25T19:29:03.352 DEBUG:teuthology.orchestra.run.smithi006:> rm -fr /tmp/tmp.qmRABefvBd 2024-03-25T19:29:03.366 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621303/data/mon.b.tgz 2024-03-25T19:29:03.366 DEBUG:teuthology.orchestra.run.smithi192:> mktemp 2024-03-25T19:29:03.382 INFO:teuthology.orchestra.run.smithi192.stdout:/tmp/tmp.Vgy3S7gEmQ 2024-03-25T19:29:03.382 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.Vgy3S7gEmQ 2024-03-25T19:29:03.485 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0666 /tmp/tmp.Vgy3S7gEmQ 2024-03-25T19:29:03.570 DEBUG:teuthology.orchestra.remote:smithi192:/tmp/tmp.Vgy3S7gEmQ is 155KB 2024-03-25T19:29:03.594 DEBUG:teuthology.orchestra.run.smithi192:> rm -fr /tmp/tmp.Vgy3S7gEmQ 2024-03-25T19:29:03.607 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T19:29:03.607 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T19:29:03.610 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T19:29:03.993 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:29:04.175 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T19:29:04.177 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621303/remote/smithi006/crash 2024-03-25T19:29:04.178 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T19:29:04.204 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621303/remote/smithi192/crash 2024-03-25T19:29:04.205 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T19:29:04.231 INFO:tasks.ceph:Compressing logs... 2024-03-25T19:29:04.231 DEBUG:teuthology.orchestra.run.smithi006:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T19:29:04.249 DEBUG:teuthology.orchestra.run.smithi192:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T19:29:04.323 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T19:29:04.323 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34858.log 2024-03-25T19:29:04.323 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T19:29:04.324 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-client.admin.34858.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T19:29:04.324 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T19:29:04.324 INFO:teuthology.orchestra.run.smithi192.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T19:29:04.324 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.34858.log.gz 2024-03-25T19:29:04.324 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T19:29:04.324 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/valgrind/osd.5.log: /var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T19:29:04.324 INFO:teuthology.orchestra.run.smithi192.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T19:29:04.324 INFO:teuthology.orchestra.run.smithi192.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T19:29:04.324 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35109.log 2024-03-25T19:29:04.324 INFO:teuthology.orchestra.run.smithi192.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T19:29:04.324 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35058.log 2024-03-25T19:29:04.324 INFO:teuthology.orchestra.run.smithi192.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T19:29:04.324 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-client.admin.35109.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T19:29:04.324 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-client.admin.35058.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T19:29:04.324 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35109.log.gz 2024-03-25T19:29:04.324 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34978.log 2024-03-25T19:29:04.325 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35058.log.gz 2024-03-25T19:29:04.325 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T19:29:04.325 INFO:teuthology.orchestra.run.smithi192.stderr: 88.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T19:29:04.325 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-client.admin.34978.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T19:29:04.325 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-osd.7.log: 87.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.34978.log.gz 2024-03-25T19:29:04.325 INFO:teuthology.orchestra.run.smithi192.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.34898.log 2024-03-25T19:29:04.325 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T19:29:04.325 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-osd.5.log: /var/log/ceph/ceph-client.admin.34898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34818.log 2024-03-25T19:29:04.325 INFO:teuthology.orchestra.run.smithi192.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.34898.log.gz 2024-03-25T19:29:04.325 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T19:29:04.325 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-client.admin.34818.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T19:29:04.325 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34938.log 2024-03-25T19:29:04.325 INFO:teuthology.orchestra.run.smithi192.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.34818.log.gz 2024-03-25T19:29:04.325 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35018.log 2024-03-25T19:29:04.325 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-client.admin.34938.log: 86.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T19:29:04.326 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34938.log.gz 2024-03-25T19:29:04.326 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-client.admin.35018.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35018.log.gz 2024-03-25T19:29:04.326 INFO:teuthology.orchestra.run.smithi192.stderr: 93.9% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T19:29:04.329 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T19:29:04.329 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42198.log 2024-03-25T19:29:04.329 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40011.log 2024-03-25T19:29:04.329 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42276.log 2024-03-25T19:29:04.329 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40831.log 2024-03-25T19:29:04.329 INFO:teuthology.orchestra.run.smithi006.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.42198.log.gz 2024-03-25T19:29:04.329 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39660.log 2024-03-25T19:29:04.329 INFO:teuthology.orchestra.run.smithi006.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40011.log.gz 2024-03-25T19:29:04.330 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42276.log: /var/log/ceph/ceph-client.admin.40831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40950.log 2024-03-25T19:29:04.330 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39504.log 2024-03-25T19:29:04.330 INFO:teuthology.orchestra.run.smithi006.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42276.log.gz 2024-03-25T19:29:04.330 INFO:teuthology.orchestra.run.smithi006.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40831.log.gz 2024-03-25T19:29:04.330 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40168.log 2024-03-25T19:29:04.330 INFO:teuthology.orchestra.run.smithi006.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39660.log.gz 2024-03-25T19:29:04.330 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40246.log 2024-03-25T19:29:04.330 INFO:teuthology.orchestra.run.smithi006.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40950.log.gz 2024-03-25T19:29:04.330 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40168.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39504.log.gz 2024-03-25T19:29:04.330 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T19:29:04.330 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40246.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T19:29:04.330 INFO:teuthology.orchestra.run.smithi006.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40168.log.gz 2024-03-25T19:29:04.330 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T19:29:04.330 INFO:teuthology.orchestra.run.smithi006.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40246.log.gz 2024-03-25T19:29:04.330 INFO:teuthology.orchestra.run.smithi006.stderr: 69.5% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T19:29:04.331 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T19:29:04.331 INFO:teuthology.orchestra.run.smithi006.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T19:29:04.331 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T19:29:04.331 INFO:teuthology.orchestra.run.smithi006.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T19:29:04.331 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T19:29:04.331 INFO:teuthology.orchestra.run.smithi006.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T19:29:04.331 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41145.log 2024-03-25T19:29:04.331 INFO:teuthology.orchestra.run.smithi006.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T19:29:04.331 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40870.log 2024-03-25T19:29:04.331 INFO:teuthology.orchestra.run.smithi006.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T19:29:04.331 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41847.log 2024-03-25T19:29:04.331 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42120.log 2024-03-25T19:29:04.331 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40870.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41145.log.gz 2024-03-25T19:29:04.331 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35450.log 2024-03-25T19:29:04.332 INFO:teuthology.orchestra.run.smithi006.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40870.log.gz 2024-03-25T19:29:04.332 INFO:teuthology.orchestra.run.smithi006.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41847.log.gz 2024-03-25T19:29:04.332 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39308.log 2024-03-25T19:29:04.332 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42120.log: /var/log/ceph/ceph-client.admin.35450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39072.log 2024-03-25T19:29:04.332 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42042.log 2024-03-25T19:29:04.332 INFO:teuthology.orchestra.run.smithi006.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.42120.log.gz 2024-03-25T19:29:04.332 INFO:teuthology.orchestra.run.smithi006.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.35450.log.gz 2024-03-25T19:29:04.332 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39269.log 2024-03-25T19:29:04.332 INFO:teuthology.orchestra.run.smithi006.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39308.log.gz 2024-03-25T19:29:04.332 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42042.log: 86.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.39072.log.gz 2024-03-25T19:29:04.332 INFO:teuthology.orchestra.run.smithi006.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41106.log 2024-03-25T19:29:04.332 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39269.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.42042.log.gz 2024-03-25T19:29:04.332 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41028.log 2024-03-25T19:29:04.332 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41652.log 2024-03-25T19:29:04.332 INFO:teuthology.orchestra.run.smithi006.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39269.log.gz 2024-03-25T19:29:04.332 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41028.log: gzip 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41106.log.gz -5 2024-03-25T19:29:04.332 INFO:teuthology.orchestra.run.smithi006.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39152.log 2024-03-25T19:29:04.333 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42081.log 2024-03-25T19:29:04.333 INFO:teuthology.orchestra.run.smithi006.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41028.log.gz 2024-03-25T19:29:04.333 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39152.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29282.log 2024-03-25T19:29:04.333 INFO:teuthology.orchestra.run.smithi006.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41652.log.gz 2024-03-25T19:29:04.333 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42081.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39152.log.gz 2024-03-25T19:29:04.333 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41379.log 2024-03-25T19:29:04.333 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph.tmp-client.admin.29282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41535.log 2024-03-25T19:29:04.333 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29282.log.gz 2024-03-25T19:29:04.333 INFO:teuthology.orchestra.run.smithi006.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.42081.log.gz 2024-03-25T19:29:04.333 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41340.log 2024-03-25T19:29:04.333 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41535.log: gzip 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41379.log.gz 2024-03-25T19:29:04.333 INFO:teuthology.orchestra.run.smithi006.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42237.log 2024-03-25T19:29:04.333 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41340.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T19:29:04.333 INFO:teuthology.orchestra.run.smithi006.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41535.log.gz 2024-03-25T19:29:04.334 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41691.log 2024-03-25T19:29:04.334 INFO:teuthology.orchestra.run.smithi006.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41340.log.gz 2024-03-25T19:29:04.334 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37626.log 2024-03-25T19:29:04.334 INFO:teuthology.orchestra.run.smithi006.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42237.log.gz 2024-03-25T19:29:04.334 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39621.log 2024-03-25T19:29:04.334 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.37626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40051.log 2024-03-25T19:29:04.334 INFO:teuthology.orchestra.run.smithi006.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41691.log.gz 2024-03-25T19:29:04.334 INFO:teuthology.orchestra.run.smithi006.stderr: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T19:29:04.334 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40363.log 2024-03-25T19:29:04.334 INFO:teuthology.orchestra.run.smithi006.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37626.log.gz 2024-03-25T19:29:04.334 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42159.log 2024-03-25T19:29:04.334 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40051.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39621.log.gz 2024-03-25T19:29:04.334 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39972.log 2024-03-25T19:29:04.334 INFO:teuthology.orchestra.run.smithi006.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40051.log.gz 2024-03-25T19:29:04.334 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42159.log: gzip 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40363.log.gz -5 2024-03-25T19:29:04.334 INFO:teuthology.orchestra.run.smithi006.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40558.log 2024-03-25T19:29:04.335 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36584.log 2024-03-25T19:29:04.335 INFO:teuthology.orchestra.run.smithi006.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.42159.log.gz 2024-03-25T19:29:04.335 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40558.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39972.log.gz 2024-03-25T19:29:04.335 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41418.log 2024-03-25T19:29:04.335 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.36584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40714.log 2024-03-25T19:29:04.335 INFO:teuthology.orchestra.run.smithi006.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40558.log.gz 2024-03-25T19:29:04.335 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41418.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T19:29:04.335 INFO:teuthology.orchestra.run.smithi006.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.36584.log.gz 2024-03-25T19:29:04.335 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40714.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41418.log.gz 2024-03-25T19:29:04.335 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40402.log 2024-03-25T19:29:04.335 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39894.log 2024-03-25T19:29:04.335 INFO:teuthology.orchestra.run.smithi006.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40714.log.gz 2024-03-25T19:29:04.335 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41184.log 2024-03-25T19:29:04.335 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39894.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40402.log.gz 2024-03-25T19:29:04.335 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40480.log 2024-03-25T19:29:04.336 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42003.log 2024-03-25T19:29:04.336 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41184.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39894.log.gz 2024-03-25T19:29:04.336 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40675.log 2024-03-25T19:29:04.336 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42003.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41184.log.gz 2024-03-25T19:29:04.336 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41730.log 2024-03-25T19:29:04.336 INFO:teuthology.orchestra.run.smithi006.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40480.log.gz 2024-03-25T19:29:04.336 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40675.log: 87.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.42003.log.gz 2024-03-25T19:29:04.336 INFO:teuthology.orchestra.run.smithi006.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39347.log 2024-03-25T19:29:04.336 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40324.log 2024-03-25T19:29:04.336 INFO:teuthology.orchestra.run.smithi006.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40675.log.gz 2024-03-25T19:29:04.336 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41925.log 2024-03-25T19:29:04.336 INFO:teuthology.orchestra.run.smithi006.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41730.log.gz 2024-03-25T19:29:04.336 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40636.log 2024-03-25T19:29:04.337 INFO:teuthology.orchestra.run.smithi006.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39347.log.gz 2024-03-25T19:29:04.337 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41925.log: 86.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.40324.log.gz 2024-03-25T19:29:04.337 INFO:teuthology.orchestra.run.smithi006.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41613.log 2024-03-25T19:29:04.337 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40441.log 2024-03-25T19:29:04.337 INFO:teuthology.orchestra.run.smithi006.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41925.log.gz 2024-03-25T19:29:04.337 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40989.log 2024-03-25T19:29:04.337 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40441.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40636.log.gz 2024-03-25T19:29:04.337 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T19:29:04.337 INFO:teuthology.orchestra.run.smithi006.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41613.log.gz 2024-03-25T19:29:04.337 INFO:teuthology.orchestra.run.smithi006.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40441.log.gz 2024-03-25T19:29:04.337 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5/var/log/ceph/ceph-client.admin.40989.log: --verbose -- /var/log/ceph/ceph-client.admin.39191.log 2024-03-25T19:29:04.337 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39933.log 2024-03-25T19:29:04.337 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39191.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40989.log.gz 2024-03-25T19:29:04.337 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41808.log 2024-03-25T19:29:04.338 INFO:teuthology.orchestra.run.smithi006.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39191.log.gz 2024-03-25T19:29:04.338 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40753.log 2024-03-25T19:29:04.338 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41886.log 2024-03-25T19:29:04.338 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41808.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39933.log.gz 2024-03-25T19:29:04.338 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35712.log 2024-03-25T19:29:04.338 INFO:teuthology.orchestra.run.smithi006.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41808.log.gz 2024-03-25T19:29:04.338 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40911.log 2024-03-25T19:29:04.338 INFO:teuthology.orchestra.run.smithi006.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40753.log.gz 2024-03-25T19:29:04.338 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.35712.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41886.log.gz 2024-03-25T19:29:04.338 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41496.log 2024-03-25T19:29:04.338 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41262.log 2024-03-25T19:29:04.338 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35712.log.gz 2024-03-25T19:29:04.338 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39777.log 2024-03-25T19:29:04.339 INFO:teuthology.orchestra.run.smithi006.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40911.log.gz 2024-03-25T19:29:04.339 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41262.log: gzip -5 --verbose -- 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41496.log.gz 2024-03-25T19:29:04.339 INFO:teuthology.orchestra.run.smithi006.stderr: /var/log/ceph/ceph-client.admin.40792.log 2024-03-25T19:29:04.339 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35555.log 2024-03-25T19:29:04.339 INFO:teuthology.orchestra.run.smithi006.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41262.log.gz 2024-03-25T19:29:04.339 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39111.log 2024-03-25T19:29:04.339 INFO:teuthology.orchestra.run.smithi006.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39777.log.gz 2024-03-25T19:29:04.339 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.35555.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T19:29:04.339 INFO:teuthology.orchestra.run.smithi006.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40792.log.gz 2024-03-25T19:29:04.339 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41223.log 2024-03-25T19:29:04.339 INFO:teuthology.orchestra.run.smithi006.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.35555.log.gz 2024-03-25T19:29:04.339 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35146.log 2024-03-25T19:29:04.339 INFO:teuthology.orchestra.run.smithi006.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39111.log.gz 2024-03-25T19:29:04.339 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41457.log 2024-03-25T19:29:04.339 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.35146.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41223.log.gz 2024-03-25T19:29:04.340 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39387.log 2024-03-25T19:29:04.340 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40285.log 2024-03-25T19:29:04.340 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39387.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.35146.log.gz 2024-03-25T19:29:04.340 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40597.log 2024-03-25T19:29:04.340 INFO:teuthology.orchestra.run.smithi006.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41457.log.gz 2024-03-25T19:29:04.340 INFO:teuthology.orchestra.run.smithi006.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39387.log.gz 2024-03-25T19:29:04.340 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39543.log 2024-03-25T19:29:04.340 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40285.log: /var/log/ceph/ceph-client.admin.40597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40090.log 2024-03-25T19:29:04.340 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39816.log 2024-03-25T19:29:04.340 INFO:teuthology.orchestra.run.smithi006.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40285.log.gz 2024-03-25T19:29:04.340 INFO:teuthology.orchestra.run.smithi006.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40597.log.gz 2024-03-25T19:29:04.340 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40090.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39543.log.gz 2024-03-25T19:29:04.340 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41067.log 2024-03-25T19:29:04.341 INFO:teuthology.orchestra.run.smithi006.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40090.log.gz 2024-03-25T19:29:04.341 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39230.log 2024-03-25T19:29:04.341 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39582.log 2024-03-25T19:29:04.341 INFO:teuthology.orchestra.run.smithi006.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39816.log.gz 2024-03-25T19:29:04.341 INFO:teuthology.orchestra.run.smithi006.stderr:gzip/var/log/ceph/ceph-client.admin.39230.log: -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T19:29:04.341 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39582.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41067.log.gz 2024-03-25T19:29:04.341 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39465.log 2024-03-25T19:29:04.341 INFO:teuthology.orchestra.run.smithi006.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39230.log.gz 2024-03-25T19:29:04.341 INFO:teuthology.orchestra.run.smithi006.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39582.log.gz 2024-03-25T19:29:04.341 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39855.log 2024-03-25T19:29:04.341 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41769.log 2024-03-25T19:29:04.341 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38855.log 2024-03-25T19:29:04.341 INFO:teuthology.orchestra.run.smithi006.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39465.log.gz 2024-03-25T19:29:04.341 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40207.log 2024-03-25T19:29:04.341 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41769.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39855.log.gz/var/log/ceph/ceph-client.admin.38855.log: 2024-03-25T19:29:04.342 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T19:29:04.342 INFO:teuthology.orchestra.run.smithi006.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41769.log.gz 2024-03-25T19:29:04.342 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40207.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38855.log.gz 2024-03-25T19:29:04.342 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39426.log 2024-03-25T19:29:04.342 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40207.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T19:29:04.342 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-03-25T19:29:04.342 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39426.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39426.log.gz 2024-03-25T19:29:04.342 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41574.log 2024-03-25T19:29:04.342 INFO:teuthology.orchestra.run.smithi006.stderr: 92.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T19:29:04.342 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42316.log 2024-03-25T19:29:04.342 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40129.log 2024-03-25T19:29:04.342 INFO:teuthology.orchestra.run.smithi006.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41574.log.gz 2024-03-25T19:29:04.342 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40519.log 2024-03-25T19:29:04.342 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40129.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.42316.log.gz 2024-03-25T19:29:04.343 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41964.log 2024-03-25T19:29:04.343 INFO:teuthology.orchestra.run.smithi006.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40129.log.gz 2024-03-25T19:29:04.343 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40519.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40519.log.gz 2024-03-25T19:29:04.343 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41301.log 2024-03-25T19:29:04.343 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39738.log 2024-03-25T19:29:04.343 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41301.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41964.log.gz 2024-03-25T19:29:04.343 INFO:teuthology.orchestra.run.smithi006.stderr: 86.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39699.log 2024-03-25T19:29:04.343 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41301.log.gz 2024-03-25T19:29:04.343 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42355.log 2024-03-25T19:29:04.343 INFO:teuthology.orchestra.run.smithi006.stderr: 87.1%/var/log/ceph/ceph-client.admin.39699.log: -- replaced with /var/log/ceph/ceph-client.admin.39738.log.gz 2024-03-25T19:29:04.343 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T19:29:04.343 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42355.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39699.log.gz 2024-03-25T19:29:04.343 INFO:teuthology.orchestra.run.smithi006.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.42355.log.gz 2024-03-25T19:29:04.375 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-mon.c.log: 92.5% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T19:29:05.127 INFO:teuthology.orchestra.run.smithi006.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T19:29:05.194 INFO:teuthology.orchestra.run.smithi192.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T19:29:05.218 INFO:teuthology.orchestra.run.smithi192.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T19:29:05.396 INFO:teuthology.orchestra.run.smithi006.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T19:29:05.418 INFO:teuthology.orchestra.run.smithi006.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T19:29:05.464 INFO:teuthology.orchestra.run.smithi192.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T19:29:05.492 INFO:teuthology.orchestra.run.smithi192.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T19:29:05.528 INFO:teuthology.orchestra.run.smithi006.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T19:29:09.294 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T19:29:09.295 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-25T19:29:09.295 INFO:teuthology.orchestra.run.smithi192.stderr:real 0m5.003s 2024-03-25T19:29:09.295 INFO:teuthology.orchestra.run.smithi192.stderr:user 0m8.854s 2024-03-25T19:29:09.295 INFO:teuthology.orchestra.run.smithi192.stderr:sys 0m0.308s 2024-03-25T19:29:09.466 INFO:teuthology.orchestra.run.smithi006.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T19:29:09.466 INFO:teuthology.orchestra.run.smithi006.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T19:29:09.468 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-03-25T19:29:09.468 INFO:teuthology.orchestra.run.smithi006.stderr:real 0m5.208s 2024-03-25T19:29:09.468 INFO:teuthology.orchestra.run.smithi006.stderr:user 0m14.323s 2024-03-25T19:29:09.469 INFO:teuthology.orchestra.run.smithi006.stderr:sys 0m0.510s 2024-03-25T19:29:09.469 INFO:tasks.ceph:Archiving logs... 2024-03-25T19:29:09.469 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621303/remote/smithi006/log 2024-03-25T19:29:09.470 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T19:29:11.841 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621303/remote/smithi192/log 2024-03-25T19:29:11.842 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T19:29:13.365 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:29:13.570 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=7e6017414b0644c6b85fcb491bff037a Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:29:13.574 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T19:29:13.584 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T19:29:13.601 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T19:29:13.601 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T19:29:13.607 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T19:29:13.695 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T19:29:13.703 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T19:29:13.704 DEBUG:teuthology.orchestra.run.smithi006:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T19:29:13.706 DEBUG:teuthology.orchestra.run.smithi192:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T19:29:13.720 INFO:teuthology.orchestra.run.smithi006.stderr:bash: line 1: ntpq: command not found 2024-03-25T19:29:13.723 INFO:teuthology.orchestra.run.smithi192.stderr:bash: line 1: ntpq: command not found 2024-03-25T19:29:13.725 INFO:teuthology.orchestra.run.smithi006.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T19:29:13.725 INFO:teuthology.orchestra.run.smithi006.stdout:=============================================================================== 2024-03-25T19:29:13.725 INFO:teuthology.orchestra.run.smithi006.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 6 -128us[ -128us] +/- 74ms 2024-03-25T19:29:13.726 INFO:teuthology.orchestra.run.smithi006.stdout:^* hv02.front.sepia.ceph.com 3 6 377 7 -390us[ -386us] +/- 59ms 2024-03-25T19:29:13.726 INFO:teuthology.orchestra.run.smithi006.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 9 +776us[ +780us] +/- 79ms 2024-03-25T19:29:13.726 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T19:29:13.728 INFO:teuthology.orchestra.run.smithi192.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T19:29:13.728 INFO:teuthology.orchestra.run.smithi192.stdout:=============================================================================== 2024-03-25T19:29:13.728 INFO:teuthology.orchestra.run.smithi192.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 11 -122us[ -110us] +/- 74ms 2024-03-25T19:29:13.728 INFO:teuthology.orchestra.run.smithi192.stdout:^* hv02.front.sepia.ceph.com 3 6 377 7 -392us[ -380us] +/- 59ms 2024-03-25T19:29:13.728 INFO:teuthology.orchestra.run.smithi192.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 12 +748us[ +760us] +/- 79ms 2024-03-25T19:29:13.729 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T19:29:13.729 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T19:29:13.737 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T19:29:13.737 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T19:29:13.746 DEBUG:teuthology.orchestra.run.smithi006:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T19:29:13.839 DEBUG:teuthology.orchestra.run.smithi192:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T19:29:13.911 DEBUG:teuthology.orchestra.run.smithi006:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T19:29:13.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:29:13.942 DEBUG:teuthology.orchestra.run.smithi192:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T19:29:13.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:29:13.974 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T19:29:13.983 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T19:29:13.992 INFO:teuthology.task.internal:Duration was 6502.301060 seconds 2024-03-25T19:29:13.992 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T19:29:14.000 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T19:29:14.001 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T19:29:14.003 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T19:29:14.045 INFO:teuthology.orchestra.run.smithi006.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T19:29:14.095 INFO:teuthology.orchestra.run.smithi192.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T19:29:14.428 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T19:29:14.428 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi006.front.sepia.ceph.com 2024-03-25T19:29:14.428 DEBUG:teuthology.orchestra.run.smithi006:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T19:29:14.449 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi192.front.sepia.ceph.com 2024-03-25T19:29:14.449 DEBUG:teuthology.orchestra.run.smithi192:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T19:29:14.482 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T19:29:14.482 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T19:29:14.492 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T19:29:14.555 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T19:29:14.556 DEBUG:teuthology.orchestra.run.smithi006:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T19:29:14.561 DEBUG:teuthology.orchestra.run.smithi192:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T19:29:14.703 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T19:29:14.713 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T19:29:14.713 DEBUG:teuthology.orchestra.run.smithi006:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T19:29:14.740 DEBUG:teuthology.orchestra.run.smithi192:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T19:29:14.767 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T19:29:14.777 DEBUG:teuthology.orchestra.run.smithi006:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:29:14.783 DEBUG:teuthology.orchestra.run.smithi192:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:29:14.806 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = core 2024-03-25T19:29:14.833 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = core 2024-03-25T19:29:14.855 DEBUG:teuthology.orchestra.run.smithi006:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:29:14.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:29:14.883 DEBUG:teuthology.orchestra.run.smithi192:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:29:14.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:29:14.910 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T19:29:14.919 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T19:29:14.920 DEBUG:teuthology.misc:Transferring archived files from smithi006:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621303/remote/smithi006 2024-03-25T19:29:14.920 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T19:29:14.958 DEBUG:teuthology.misc:Transferring archived files from smithi192:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621303/remote/smithi192 2024-03-25T19:29:14.958 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T19:29:14.991 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T19:29:14.992 DEBUG:teuthology.orchestra.run.smithi006:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T19:29:14.996 DEBUG:teuthology.orchestra.run.smithi192:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T19:29:15.041 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T19:29:15.051 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T19:29:15.052 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T19:29:15.060 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T19:29:15.060 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T19:29:15.062 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T19:29:15.074 INFO:teuthology.orchestra.run.smithi006.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 19:29 /home/ubuntu/cephtest 2024-03-25T19:29:15.096 INFO:teuthology.orchestra.run.smithi192.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 19:29 /home/ubuntu/cephtest 2024-03-25T19:29:15.097 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T19:29:15.107 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T19:29:15.126 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T19:29:15.150 DEBUG:teuthology.nuke:shortname: smithi006 2024-03-25T19:29:15.150 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T19:29:15.175 DEBUG:teuthology.nuke:shortname: smithi192 2024-03-25T19:29:15.176 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T19:29:15.179 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi006.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621303', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 17:30:57.355892', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJzS5jxOxGYBkmXRwUv79FYTq0F6mAdD0VcTyIyO6APFUyBkiyxdulhH119XzUV6ATQR9i1XYNrHv24NcvH/MVc='} 2024-03-25T19:29:15.202 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621303', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 17:30:57.354215', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNSz2WXJW1fvNy/bJE8JQNd9/IgUiV02UxqYAAg6OOkO/eljBodBu+CP26wgr8RmCgIKY3bFpC1hVE40eVVCvdc='} 2024-03-25T19:29:15.225 INFO:teuthology.orchestra.console.smithi006:Power off 2024-03-25T19:29:15.226 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T19:29:15.249 DEBUG:teuthology.orchestra.console.smithi006:power off output: Chassis Power Control: Down/Off 2024-03-25T19:29:15.249 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:29:15.264 INFO:teuthology.orchestra.console.smithi192:Power off 2024-03-25T19:29:15.264 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T19:29:15.274 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is on 2024-03-25T19:29:15.287 DEBUG:teuthology.orchestra.console.smithi192:power off output: Chassis Power Control: Down/Off 2024-03-25T19:29:15.288 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:29:15.312 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is on 2024-03-25T19:29:19.275 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:29:19.313 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:29:19.388 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is on 2024-03-25T19:29:19.422 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is on 2024-03-25T19:29:23.393 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:29:23.424 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:29:23.503 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is on 2024-03-25T19:29:23.533 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is off 2024-03-25T19:29:23.634 INFO:teuthology.orchestra.console.smithi192:Power off completed 2024-03-25T19:29:27.506 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:29:27.616 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is off 2024-03-25T19:29:27.717 INFO:teuthology.orchestra.console.smithi006:Power off completed 2024-03-25T19:29:27.821 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 6502.301060438156 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=7e6017414b0644c6b85fcb491bff037a status: fail success: false 2024-03-25T19:29:27.821 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T19:29:27.897 INFO:teuthology.run:FAIL